pipefunc.map.adaptive_scheduler module

Provides adaptive_scheduler integration for pipefunc.

class pipefunc.map.adaptive_scheduler.AdaptiveSchedulerDetails(learners: list[SequenceLearner], fnames: list[Path], dependencies: dict[int, list[int]], nodes: tuple[int | None | Callable[[], int | None], ...] | None, cores_per_node: tuple[int | None | Callable[[], int | None], ...] | None, extra_scheduler: tuple[list[str] | Callable[[], list[str]], ...] | None, partition: tuple[str | None | Callable[[], str | None], ...] | None, executor_type: tuple[EXECUTOR_TYPES | Callable[[], EXECUTOR_TYPES], ...] | None = None)

Bases: NamedTuple

Details for the adaptive scheduler.

learners: list[SequenceLearner]

Alias for field number 0

fnames: list[Path]

Alias for field number 1

dependencies: dict[int, list[int]]

Alias for field number 2

nodes: tuple[int | None | Callable[[], int | None], ...] | None

Alias for field number 3

cores_per_node: tuple[int | None | Callable[[], int | None], ...] | None

Alias for field number 4

extra_scheduler: tuple[list[str] | Callable[[], list[str]], ...] | None

Alias for field number 5

partition: tuple[str | None | Callable[[], str | None], ...] | None

Alias for field number 6

executor_type: tuple[EXECUTOR_TYPES | Callable[[], EXECUTOR_TYPES], ...] | None

Alias for field number 7

kwargs()

Get keyword arguments for adaptive_scheduler.slurm_run.

Return type:

dict[str, Any]

Examples

>>> learners = pipefunc.map.adaptive.create_learners(pipeline, ...)
>>> info = learners.to_slurm_run(...)
>>> kwargs = info.kwargs()
>>> adaptive_scheduler.slurm_run(**kwargs)

run_manager(kwargs)

Get a RunManager for the adaptive scheduler.

Return type:

RunManager

pipefunc.map.adaptive_scheduler.slurm_run_setup(learners_dict, default_resources=None, *, ignore_resources=False)

Set up the arguments for adaptive_scheduler.slurm_run.

Return type:

AdaptiveSchedulerDetails