AWSBatchRunner#
- class AWSBatchRunner[source]#
Bases:
Runner
Runs pipelines remotely using AWS Batch.
Requires Everett configuration of form:
[AWS_BATCH] cpu_job_queue= cpu_job_def= gpu_job_queue= gpu_job_def= attempts=
- __init__()#
Methods
__init__
()build_cmd
(cfg_json_uri, pipeline, commands)Get the split_ind for the process.
run
(cfg_json_uri, pipeline, commands[, ...])Run commands in a Pipeline using a serialized PipelineConfig.
run_command
(cmd[, job_name, debug, ...])Submit a command as a job to AWS Batch.
- build_cmd(cfg_json_uri: str, pipeline: Pipeline, commands: List[str], num_splits: int = 1, pipeline_run_name: str = 'raster-vision') Tuple[List[str], Dict[str, Any]] [source]#
- get_split_ind() int [source]#
Get the split_ind for the process.
For split commands, the split_ind determines which split of work to perform within the current OS process. The CLI has a –split-ind option, but some runners may have their own means of communicating the split_ind, and this method should be overridden in such cases. If this method returns None, then the –split-ind option will be used. If both are null, then it won’t be possible to run the command.
- Return type
- run(cfg_json_uri: str, pipeline: Pipeline, commands: List[str], num_splits: int = 1, pipeline_run_name: str = 'raster-vision')[source]#
Run commands in a Pipeline using a serialized PipelineConfig.
- run_command(cmd: List[str], job_name: Optional[str] = None, debug: bool = False, attempts: int = 1, parent_job_ids: Optional[List[str]] = None, num_array_jobs: Optional[int] = None, use_gpu: bool = False, job_queue: Optional[str] = None, job_def: Optional[str] = None, **kwargs) str [source]#
Submit a command as a job to AWS Batch.
- Parameters
cmd (List[str]) – Command to run in the Docker container for the remote job as list of strings.
job_name (Optional[str]) – Optional job name. If None, is set to “raster-vision-<uuid>”.
debug (bool) – If True, run the command using a ptvsd wrapper which sets up a remote VS Code Python debugger server. Defaults to False.
attempts (int) – The number of times to try running the command which is useful in case of failure. Defaults to 5.
parent_job_ids (Optional[List[str]]) – Optional list of parent Batch job IDs. The job created by this will only run after the parent jobs complete successfully. Defaults to None.
num_array_jobs (Optional[int]) – If set, make this a Batch array job with size equal to num_array_jobs. Defaults to None.
use_gpu (bool) – If True, run the job in a GPU-enabled queue. Defaults to False.
job_queue (Optional[str]) – If set, use this job queue. Default to None.
job_def (Optional[str]) – If set, use this job definition. Default to None.
**kwargs – Any other kwargs to pass to Batch when submitting job.
- Return type