Pipeline#

class Pipeline[source]#

Bases: object

A pipeline of commands to run sequentially.

This is an abstraction over a sequence of commands. Each command is represented by a method. This base class has two test commands, and new pipelines should be created by subclassing this.

Note that any split command methods should have the following signature: def my_command(self, split_ind: int = 0, num_splits: int = 1) The num_splits represents how many parallel jobs should be created, and the split_ind is the index of the current job within that set.

commands#

command names listed in the order in which they should run

Type

List[str]

split_commands#

names of commands that can be split and run in parallel

Type

List[str]

gpu_commands#

names of commands that should be executed on GPUs if available

Type

List[str]

Attributes

__init__(config: PipelineConfig, tmp_dir: str)[source]#

Constructor

Parameters
  • config (PipelineConfig) – the configuration of this pipeline

  • tmp_dir (str) – the root any temporary directories created by running this pipeline

Methods

__init__(config, tmp_dir)

Constructor

test_cpu([split_ind, num_splits])

A command to test the ability to run split jobs on CPU.

test_gpu()

A command to test the ability to run on GPU.

__init__(config: PipelineConfig, tmp_dir: str)[source]#

Constructor

Parameters
  • config (PipelineConfig) – the configuration of this pipeline

  • tmp_dir (str) – the root any temporary directories created by running this pipeline

test_cpu(split_ind: int = 0, num_splits: int = 1)[source]#

A command to test the ability to run split jobs on CPU.

Parameters
  • split_ind (int) –

  • num_splits (int) –

test_gpu()[source]#

A command to test the ability to run on GPU.

commands: List[str] = ['test_cpu', 'test_gpu']#
gpu_commands: List[str] = ['test_gpu']#
split_commands: List[str] = ['test_cpu']#