AWSBatchRunner#

class AWSBatchRunner[source]#

Bases: Runner

Runs pipelines remotely using AWS Batch.

Requires Everett configuration of form:

[AWS_BATCH]
cpu_job_queue=
cpu_job_def=
gpu_job_queue=
gpu_job_def=
attempts=
__init__()#

Methods

__init__()

build_cmd(cfg_json_uri, pipeline, commands)

get_split_ind()

Get the split_ind for the process.

run(cfg_json_uri, pipeline, commands[, ...])

Run commands in a Pipeline using a serialized PipelineConfig.

run_command(cmd[, job_name, debug, ...])

Submit a command as a job to AWS Batch.

build_cmd(cfg_json_uri: str, pipeline: Pipeline, commands: List[str], num_splits: int = 1, pipeline_run_name: str = 'raster-vision') Tuple[List[str], Dict[str, Any]][source]#
Parameters
  • cfg_json_uri (str) –

  • pipeline (Pipeline) –

  • commands (List[str]) –

  • num_splits (int) –

  • pipeline_run_name (str) –

Return type

Tuple[List[str], Dict[str, Any]]

get_split_ind() int[source]#

Get the split_ind for the process.

For split commands, the split_ind determines which split of work to perform within the current OS process. The CLI has a –split-ind option, but some runners may have their own means of communicating the split_ind, and this method should be overridden in such cases. If this method returns None, then the –split-ind option will be used. If both are null, then it won’t be possible to run the command.

Return type

int

run(cfg_json_uri: str, pipeline: Pipeline, commands: List[str], num_splits: int = 1, pipeline_run_name: str = 'raster-vision')[source]#

Run commands in a Pipeline using a serialized PipelineConfig.

Parameters
  • cfg_json_uri (str) – URI of a JSON file with a serialized PipelineConfig

  • pipeline (Pipeline) – the Pipeline to run

  • commands (List[str]) – names of commands to run

  • num_splits (int) – number of splits to use for splittable commands

  • pipeline_run_name (str) –

run_command(cmd: List[str], job_name: Optional[str] = None, debug: bool = False, attempts: int = 1, parent_job_ids: Optional[List[str]] = None, num_array_jobs: Optional[int] = None, use_gpu: bool = False, job_queue: Optional[str] = None, job_def: Optional[str] = None, **kwargs) str[source]#

Submit a command as a job to AWS Batch.

Parameters
  • cmd (List[str]) – Command to run in the Docker container for the remote job as list of strings.

  • job_name (Optional[str]) – Optional job name. If None, is set to “raster-vision-<uuid>”.

  • debug (bool) – If True, run the command using a ptvsd wrapper which sets up a remote VS Code Python debugger server. Defaults to False.

  • attempts (int) – The number of times to try running the command which is useful in case of failure. Defaults to 5.

  • parent_job_ids (Optional[List[str]]) – Optional list of parent Batch job IDs. The job created by this will only run after the parent jobs complete successfully. Defaults to None.

  • num_array_jobs (Optional[int]) – If set, make this a Batch array job with size equal to num_array_jobs. Defaults to None.

  • use_gpu (bool) – If True, run the job in a GPU-enabled queue. Defaults to False.

  • job_queue (Optional[str]) – If set, use this job queue. Default to None.

  • job_def (Optional[str]) – If set, use this job definition. Default to None.

  • **kwargs – Any other kwargs to pass to Batch when submitting job.

Return type

str