Parameter Sweeps #

Have uv? ⚡

If you have uv installed, you can instantly open this page as a Jupyter notebook using opennb:

uvx --with "pipefunc[docs]" opennb pipefunc/pipefunc/docs/source/concepts/parameter-sweeps.md

This command creates an ephemeral environment with all dependencies and launches the notebook in your browser in 1 second - no manual setup needed! ✨.

Alternatively, run:

uv run https://raw.githubusercontent.com/pipefunc/pipefunc/refs/heads/main/get-notebooks.py

to download all documentation as Jupyter notebooks.

The pipefunc.sweep module provides a convenient way to contruct parameter sweeps. It was developed before pipeline.map which can perform sweep operations in parallel. However, by itself pipefunc.sweep.Sweep might still be useful for cases where you have a pipeline that has no mapspec.

from pipefunc.sweep import Sweep

combos = {
    "a": [0, 1, 2],
    "b": [0, 1, 2],
    "c": [0, 1, 2],
}
# This means a Cartesian product of all the values in the lists
# while zipping ("a", "b").
sweep = Sweep(combos, dims=[("a", "b"), "c"])
sweep.list()[:10]  # show the first 10 combinations

[{'a': 0, 'b': 0, 'c': 0},
 {'a': 0, 'b': 0, 'c': 1},
 {'a': 0, 'b': 0, 'c': 2},
 {'a': 1, 'b': 1, 'c': 0},
 {'a': 1, 'b': 1, 'c': 1},
 {'a': 1, 'b': 1, 'c': 2},
 {'a': 2, 'b': 2, 'c': 0},
 {'a': 2, 'b': 2, 'c': 1},
 {'a': 2, 'b': 2, 'c': 2}]

The function set_cache_for_sweep then enables caching for nodes in the pipeline that are expected to be executed two or more times during the parameter sweep.

from pipefunc.sweep import set_cache_for_sweep

set_cache_for_sweep(output_name, pipeline, sweep, min_executions=2, verbose=True)

We can now run the sweep using e.g.,

results = [
    pipeline.run(output_name, kwargs=combo, full_output=True) for combo in sweep.list()
]

Parameter Sweeps

Parameter Sweeps#

Parameter Sweeps #