||A manager for parallel execution of jobs.|
A manager for parallel execution of jobs.
This class should be used as a context manager.
Clean up subprocesses on object deletion.
Close the queues and join all subprocesses before the object is deleted.
Enter the context manager.
__exit__(exc_type, exc_val, exc_tb)¶
Exit the context manager.
Setup the job and results queues.
Parameters: n_parallel – (optional)
int. The number of jobs that can be run in parallel. Defaults to None in which case all but one available CPUs will be used.
Collect all the available results or wait until they become available.
- n_results –
int, number of results to wait for. If n_results ≤ 0 then all available results will be returned.
- timeout – (optional)
float, number of seconds to wait for results to appear. If None (default) then it will wait until all n_results are collected.
A list of
Resultobjects with length n_results at least.
If n_results is overestimated and timeout is None, then this method will hang forever. Therefore it is recommended that a timeout is set.
- TimeoutError – if more than timeout seconds elapse before a
- Result –
- n_results –
Dispatch the jobs for parallel execution.
This method is non-blocking.
Parameters: jobs –
List[Job]. A list of jobs to run whenever resources are available.
Although the jobs are scheduled to run immediately, the actual execution may take place after indefinite delay if the job runner is occupied with older jobs.
Exit the scheduler by closing the queues and terminating the servant process.
Interrupt the scheduler and all running jobs.
Job(task, args=(), id=<factory>, meta=None)¶
Jobclass defining an experiment as a runnable task on the local machine.
The job is defined by a callable function or a script task. In the case of the former the args will be passed directly to it upon calling. Otherwise either a module will be run as a scirpt with command line arguments or a function, attribute of the module, will be called with the args as input. In both cases a
Resultobject will be returned.
- id –
int. The job identifier. Must be unique.
- args –
dict. The arguments or keyword arguments for the callable function or script.
- task –
str, a python function to run or a file path to a python script.
- id –
SlurmJob(task, args=(), id=<factory>, meta=None, output_file=None)¶
Jobsubclass to schedule tasks on Slurm.
Runs an ‘sbatch’ command in the shell with the script.
Variables: output_file – (optional)
str. Path to the file where the executed script will dump the result file. If none is provided, a temporary file will be created.