AWSSageMakerRunner#

class AWSSageMakerRunner[source]#

Bases: Runner

Runs pipelines remotely using AWS SageMaker.

Requires Everett configuration of form:

[SAGEMAKER]
role=
cpu_image=
cpu_instance_type=
gpu_image=
gpu_instance_type=
train_image=
train_instance_type=
train_instance_count=
use_spot_instances=
spot_instance_max_wait_time=
max_run_time=
__init__()#

Methods

__init__()

build_pipeline(cfg_json_uri, pipeline, commands)

Build a SageMaker Pipeline with each command as a step within it.

build_step(pipeline, step_name, job_name, ...)

Build appropriate SageMaker pipeline step.

get_split_ind()

Get the split_ind for the process.

run(cfg_json_uri, pipeline, commands[, ...])

Run commands in a Pipeline using a serialized PipelineConfig.

run_command(cmd[, use_gpu, image_uri, ...])

Run a single command as a SageMaker processing job.
build_pipeline(cfg_json_uri: str, pipeline: Pipeline, commands: List[str], num_splits: int = 1, cmd_prefix: List[str] = ['python', '-m', 'rastervision.pipeline.cli'], pipeline_run_name: str = 'rv') → SageMakerPipeline[source]#

Build a SageMaker Pipeline with each command as a step within it.

Parameters
Return type

SageMakerPipeline

build_step(pipeline: RVPipeline, step_name: str, job_name: str, cmd: List[str], role: str, image_uri: str, instance_type: str, use_spot_instances: bool, sagemaker_session: PipelineSession, instance_count: int = 1, max_wait: int = 86400, max_run: int = 86400, **kwargs) → Union[TrainingStep, ProcessingStep][source]#

Build appropriate SageMaker pipeline step.

If step_name=='train', builds a TrainingStep. Otherwise, a ProcessingStep.

Parameters

  • pipeline (RVPipeline) –

  • step_name (str) –

  • job_name (str) –

  • cmd (List[str]) –

  • role (str) –

  • image_uri (str) –

  • instance_type (str) –

  • use_spot_instances (bool) –

  • sagemaker_session (PipelineSession) –

  • instance_count (int) –

  • max_wait (int) –

  • max_run (int) –

Return type

Union[TrainingStep, ProcessingStep]

get_split_ind() → Optional[int]#

Get the split_ind for the process.

For split commands, the split_ind determines which split of work to perform within the current OS process. The CLI has a –split-ind option, but some runners may have their own means of communicating the split_ind, and this method should be overridden in such cases. If this method returns None, then the –split-ind option will be used. If both are null, then it won’t be possible to run the command.

Return type

Optional[int]

run(cfg_json_uri: str, pipeline: Pipeline, commands: List[str], num_splits: int = 1, cmd_prefix: List[str] = ['python', '-m', 'rastervision.pipeline.cli'], pipeline_run_name: str = 'rv')[source]#

Run commands in a Pipeline using a serialized PipelineConfig.

Parameters

  • cfg_json_uri (str) – URI of a JSON file with a serialized PipelineConfig

  • pipeline (Pipeline) – the Pipeline to run

  • commands (List[str]) – names of commands to run

  • num_splits (int) – number of splits to use for splittable commands

  • cmd_prefix (List[str]) –

  • pipeline_run_name (str) –

run_command(cmd: List[str], use_gpu: bool = False, image_uri: Optional[str] = None, instance_type: Optional[str] = None, role: Optional[str] = None, job_name: Optional[str] = None, sagemaker_session: Optional[Session] = None) → None[source]#

Run a single command as a SageMaker processing job.

Parameters

  • cmd (List[str]) – The command to run.

  • use_gpu (bool) – Use the GPU instance type and image from the Everett config. This is ignored if image_uri and instance_type are provided. Defaults to False.

  • image_uri (Optional[str]) – URI of docker image to use. If not provided, will be picked up from Everett config. Defaults to None.

  • instance_type (Optional[str]) – AWS instance type to use. If not provided, will be picked up from Everett config. Defaults to None.

  • role (Optional[str]) – AWS IAM role with SageMaker permissions. If not provided, will be picked up from Everett config. Defaults to None.

  • job_name (Optional[str]) – Optional job name. Defaults to None.

  • sagemaker_session (Optional[Session]) – SageMaker session. Defaults to None.

Return type

None