hypertunity.scheduling

Summary

Scheduler	A manager for parallel execution of jobs.
Job	Default Job class defining an experiment as a runnable task on the local machine.
SlurmJob	A Job subclass to schedule tasks on Slurm.
Result	A Result class to store the output of the executed Job.

API documentation

class Scheduler(n_parallel=None)

A manager for parallel execution of jobs.

A job must be of type Job which produces a Result object upon successful completion. The scheduler maintains a job and result queues.

Notes

This class should be used as a context manager.

__del__()

Clean up subprocesses on object deletion.

Close the queues and join all subprocesses before the object is deleted.

__enter__()

Enter the context manager.

__exit__(exc_type, exc_val, exc_tb)

Exit the context manager.

__init__(n_parallel=None)

Setup the job and results queues.

Parameters:n_parallel – (optional) int. The number of jobs that can be run in parallel. Defaults to None in which case all but one available CPUs will be used.

collect(n_results, timeout=None)

Collect all the available results or wait until they become available.

Parameters:

• n_results – int, number of results to wait for. If n_results ≤ 0 then all available results will be returned.
• timeout – (optional) float, number of seconds to wait for results to appear. If None (default) then it will wait until all n_results are collected.

Returns:

A list of Result objects with length n_results at least.

Notes

If n_results is overestimated and timeout is None, then this method will hang forever. Therefore it is recommended that a timeout is set.

Raises:

• TimeoutError – if more than timeout seconds elapse before a
• Result –

dispatch(jobs)

Dispatch the jobs for parallel execution.

This method is non-blocking.

Parameters:jobs – List[Job]. A list of jobs to run whenever resources are available.

Notes

Although the jobs are scheduled to run immediately, the actual execution may take place after indefinite delay if the job runner is occupied with older jobs.

exit()

Exit the scheduler by closing the queues and terminating the servant process.

interrupt()

Interrupt the scheduler and all running jobs.

class Job(task, args=(), id=<factory>, meta=None)

Default Job class defining an experiment as a runnable task on the local machine.

The job is defined by a callable function or a script task. In the case of the former the args will be passed directly to it upon calling. Otherwise either a module will be run as a scirpt with command line arguments or a function, attribute of the module, will be called with the args as input. In both cases a Result object will be returned.

Variables:

• id – int. The job identifier. Must be unique.
• args – tuple or dict. The arguments or keyword arguments for the callable function or script.
• task – Callable or str, a python function to run or a file path to a python script.

class SlurmJob(task, args=(), id=<factory>, meta=None, output_file=None)

A Job subclass to schedule tasks on Slurm.

Runs an ‘sbatch’ command in the shell with the script.

Variables:output_file – (optional) str. Path to the file where the executed script will dump the result file. If none is provided, a temporary file will be created.

class Result(data, id)

A Result class to store the output of the executed Job.

It shares the same id as the job which generated it.

Variables:

• id – int. The identifier of the Result object which corresponds to the job that has been run.
• data – Any. The output data of the job.