Pipeline#
- class Pipeline[source]#
Bases:
object
A pipeline of commands to run sequentially.
This is an abstraction over a sequence of commands. Each command is represented by a method. This base class has two test commands, and new pipelines should be created by subclassing this.
Note that any split command methods should have the following signature: def my_command(self, split_ind: int = 0, num_splits: int = 1) The num_splits represents how many parallel jobs should be created, and the split_ind is the index of the current job within that set.
Attributes
- __init__(config: PipelineConfig, tmp_dir: str)[source]#
Constructor
- Parameters:
config (PipelineConfig) – the configuration of this pipeline
tmp_dir (str) – the root any temporary directories created by running this pipeline
Methods
__init__
(config, tmp_dir)Constructor
test_cpu
([split_ind, num_splits])A command to test the ability to run split jobs on CPU.
test_gpu
()A command to test the ability to run on GPU.
- __init__(config: PipelineConfig, tmp_dir: str)[source]#
Constructor
- Parameters:
config (PipelineConfig) – the configuration of this pipeline
tmp_dir (str) – the root any temporary directories created by running this pipeline