pipefunc.resources module

pipefunc.resources module#

Provides the pipefunc.resources module, containing the Resources class.

class pipefunc.resources.Resources(cpus=None, cpus_per_node=None, nodes=None, memory=None, gpus=None, time=None, partition=None, extra_args=<factory>, parallelization_mode='external')[source]#

Bases: object

A dataclass representing computational resources for a job.

Parameters:

  • cpus (int | None) – The number of CPUs required for the job. Must be a positive integer.

  • cpus_per_node (int | None) – The number of CPUs per node required for the job. Must be a positive integer.

  • nodes (int | None) – The number of nodes required for the job. Must be a positive integer.

  • memory (str | None) – The memory required for the job. Must be a valid string (e.g., '2GB', '500MB').

  • gpus (int | None) – The number of GPUs required for the job. Must be a non-negative integer.

  • time (str | None) – The time required for the job. Must be a valid string (e.g., '2:00:00', '48:00:00').

  • partition (str | None) – The partition to submit the job to.

  • extra_args (dict[str, Any]) – Extra arguments for the job. Default is an empty dictionary.

  • parallelization_mode (Literal['internal', 'external']) – Specifies how parallelization should be handled. “internal”: The function should use the resources (e.g., cpus) to handle its own parallelization. “external”: The function should operate on a single core, with parallelization managed externally. Default is “external”.

Raises:

ValueError – If any of the input parameters do not meet the specified constraints.

Notes

Examples

>>> resources = Resources(cpus=4, memory='16GB', time='2:00:00')
>>> resources.cpus
4
>>> resources.memory
'16GB'
>>> resources.time
'2:00:00'
cpus: int | None = None#
cpus_per_node: int | None = None#
nodes: int | None = None#
memory: str | None = None#
gpus: int | None = None#
time: str | None = None#
partition: str | None = None#
extra_args: dict[str, Any]#
parallelization_mode: Literal['internal', 'external'] = 'external'#
static from_dict(data)[source]#

Create a Resources instance from a dictionary.

Parameters:

data (dict[str, Any]) – A dictionary containing the input parameters for the Resources instance.

Return type:

Resources

Returns:

A Resources instance created from the input dictionary.

static maybe_from_dict(resources)[source]#

Create a Resources instance from a dictionary, if not already an instance and not None.

Return type:

Resources | Callable[[dict[str, Any]], Resources] | None

to_slurm_options()[source]#

Convert the Resources instance to SLURM options.

Returns:

A string containing the SLURM options.

Return type:

str

update(**kwargs)[source]#

Update the Resources instance with new values.

Parameters:

**kwargs (Any) – Keyword arguments specifying the attributes to update and their new values.

Return type:

Resources

Returns:

A new Resources instance with the updated values.

static combine_max(resources_list)[source]#

Combine multiple Resources instances by taking the maximum value for each attribute.

Parameters:

resources_list (list[Resources]) – A list of Resources instances to combine.

Return type:

Resources

Returns:

A new Resources instance with the maximum values from the input instances.

with_defaults(default_resources)[source]#

Combine the Resources instance with default resources.

Return type:

Resources

static maybe_with_defaults(resources, default_resources)[source]#

Combine the Resources instance with default resources, if provided.

Return type:

Resources | Callable[[dict[str, Any]], Resources] | None

dict()[source]#

Return the Resources instance as a dictionary.

Returns:

A dictionary representation of the Resources instance.

Return type:

dict

Contents