DatasetConfig#

Note

All Configs are derived from rastervision.pipeline.config.Config, which itself is a pydantic Model.

pydantic model DatasetConfig[source]#

Configure train, validation, and test splits for a dataset.

Show JSON schema
{
   "title": "DatasetConfig",
   "description": "Configure train, validation, and test splits for a dataset.",
   "type": "object",
   "properties": {
      "class_config": {
         "$ref": "#/definitions/ClassConfig"
      },
      "train_scenes": {
         "title": "Train Scenes",
         "type": "array",
         "items": {
            "$ref": "#/definitions/SceneConfig"
         }
      },
      "validation_scenes": {
         "title": "Validation Scenes",
         "type": "array",
         "items": {
            "$ref": "#/definitions/SceneConfig"
         }
      },
      "test_scenes": {
         "title": "Test Scenes",
         "default": [],
         "type": "array",
         "items": {
            "$ref": "#/definitions/SceneConfig"
         }
      },
      "scene_groups": {
         "title": "Scene Groups",
         "description": "Groupings of scenes. Should be a dict of the form: {<group-name>: Set(scene_id_1, scene_id_2, ...)}. Three groups are added by default: \"train_scenes\", \"validation_scenes\", and \"test_scenes\"",
         "default": {},
         "type": "object",
         "additionalProperties": {
            "type": "array",
            "items": {
               "type": "string"
            },
            "uniqueItems": true
         }
      },
      "type_hint": {
         "title": "Type Hint",
         "default": "dataset",
         "enum": [
            "dataset"
         ],
         "type": "string"
      }
   },
   "required": [
      "class_config",
      "train_scenes",
      "validation_scenes"
   ],
   "additionalProperties": false,
   "definitions": {
      "ClassConfig": {
         "title": "ClassConfig",
         "description": "Configure class information for a machine learning task.",
         "type": "object",
         "properties": {
            "names": {
               "title": "Names",
               "description": "Names of classes. The i-th class in this list will have class ID = i.",
               "type": "array",
               "items": {
                  "type": "string"
               }
            },
            "colors": {
               "title": "Colors",
               "description": "Colors used to visualize classes. Can be color strings accepted by matplotlib or RGB tuples. If None, a random color will be auto-generated for each class.",
               "type": "array",
               "items": {
                  "anyOf": [
                     {
                        "type": "string"
                     },
                     {
                        "type": "array",
                        "items": {}
                     }
                  ]
               }
            },
            "null_class": {
               "title": "Null Class",
               "description": "Optional name of class in `names` to use as the null class. This is used in semantic segmentation to represent the label for imagery pixels that are NODATA or that are missing a label. If None and the class names include \"null\", it will automatically be used as the null class. If None, and this Config is part of a SemanticSegmentationConfig, a null class will be added automatically.",
               "type": "string"
            },
            "type_hint": {
               "title": "Type Hint",
               "default": "class_config",
               "enum": [
                  "class_config"
               ],
               "type": "string"
            }
         },
         "required": [
            "names"
         ],
         "additionalProperties": false
      },
      "RasterTransformerConfig": {
         "title": "RasterTransformerConfig",
         "description": "Configure a :class:`.RasterTransformer`.",
         "type": "object",
         "properties": {
            "type_hint": {
               "title": "Type Hint",
               "default": "raster_transformer",
               "enum": [
                  "raster_transformer"
               ],
               "type": "string"
            }
         },
         "additionalProperties": false
      },
      "RasterSourceConfig": {
         "title": "RasterSourceConfig",
         "description": "Configure a :class:`.RasterSource`.",
         "type": "object",
         "properties": {
            "channel_order": {
               "title": "Channel Order",
               "description": "The sequence of channel indices to use when reading imagery.",
               "type": "array",
               "items": {
                  "type": "integer"
               }
            },
            "transformers": {
               "title": "Transformers",
               "default": [],
               "type": "array",
               "items": {
                  "$ref": "#/definitions/RasterTransformerConfig"
               }
            },
            "bbox": {
               "title": "Bbox",
               "description": "User-specified bbox in pixel coords in the form (ymin, xmin, ymax, xmax). Useful for cropping the raster source so that only part of the raster is read from.",
               "type": "array",
               "minItems": 4,
               "maxItems": 4,
               "items": [
                  {
                     "type": "integer"
                  },
                  {
                     "type": "integer"
                  },
                  {
                     "type": "integer"
                  },
                  {
                     "type": "integer"
                  }
               ]
            },
            "type_hint": {
               "title": "Type Hint",
               "default": "raster_source",
               "enum": [
                  "raster_source"
               ],
               "type": "string"
            }
         },
         "additionalProperties": false
      },
      "LabelSourceConfig": {
         "title": "LabelSourceConfig",
         "description": "Configure a :class:`.LabelSource`.",
         "type": "object",
         "properties": {
            "type_hint": {
               "title": "Type Hint",
               "default": "label_source",
               "enum": [
                  "label_source"
               ],
               "type": "string"
            }
         },
         "additionalProperties": false
      },
      "LabelStoreConfig": {
         "title": "LabelStoreConfig",
         "description": "Configure a :class:`.LabelStore`.",
         "type": "object",
         "properties": {
            "type_hint": {
               "title": "Type Hint",
               "default": "label_store",
               "enum": [
                  "label_store"
               ],
               "type": "string"
            }
         },
         "additionalProperties": false
      },
      "SceneConfig": {
         "title": "SceneConfig",
         "description": "Configure a :class:`.Scene` comprising raster data & labels for an AOI.\n    ",
         "type": "object",
         "properties": {
            "id": {
               "title": "Id",
               "type": "string"
            },
            "raster_source": {
               "$ref": "#/definitions/RasterSourceConfig"
            },
            "label_source": {
               "$ref": "#/definitions/LabelSourceConfig"
            },
            "label_store": {
               "$ref": "#/definitions/LabelStoreConfig"
            },
            "aoi_uris": {
               "title": "Aoi Uris",
               "description": "List of URIs of GeoJSON files that define the AOIs for the scene. Each polygon defines an AOI which is a piece of the scene that is assumed to be fully labeled and usable for training or validation. The AOIs are assumed to be in EPSG:4326 coordinates.",
               "type": "array",
               "items": {
                  "type": "string"
               }
            },
            "type_hint": {
               "title": "Type Hint",
               "default": "scene",
               "enum": [
                  "scene"
               ],
               "type": "string"
            }
         },
         "required": [
            "id",
            "raster_source"
         ],
         "additionalProperties": false
      }
   }
}

Config
  • extra: str = forbid

  • validate_assignment: bool = True

Fields
field class_config: ClassConfig [Required]#
field scene_groups: Dict[str, Set[str]] = {}#

Groupings of scenes. Should be a dict of the form: {<group-name>: Set(scene_id_1, scene_id_2, …)}. Three groups are added by default: “train_scenes”, “validation_scenes”, and “test_scenes”

field test_scenes: List[SceneConfig] = []#
field train_scenes: List[SceneConfig] [Required]#
field type_hint: Literal['dataset'] = 'dataset'#
field validation_scenes: List[SceneConfig] [Required]#
build()#

Build an instance of the corresponding type of object using this config.

For example, BackendConfig will build a Backend object. The arguments to this method will vary depending on the type of Config.

classmethod from_file(uri: str) Config#

Deserialize a Config from a JSON file, upgrading if possible.

Parameters

uri (str) – URI to load from.

Return type

Config

get_split_config(split_ind, num_splits)[source]#
recursive_validate_config()#

Recursively validate hierarchies of Configs.

This uses reflection to call validate_config on a hierarchy of Configs using a depth-first pre-order traversal.

revalidate()#

Re-validate an instantiated Config.

Runs all Pydantic validators plus self.validate_config().

Adapted from: https://github.com/samuelcolvin/pydantic/issues/1864#issuecomment-679044432

to_file(uri: str, with_rv_metadata: bool = True) None#

Save a Config to a JSON file, optionally with RV metadata.

Parameters
  • uri (str) – URI to save to.

  • with_rv_metadata (bool) – If True, inject Raster Vision metadata such as plugin_versions, so that the config can be upgraded when loaded.

Return type

None

update(pipeline=None)[source]#

Update any fields before validation.

Subclasses should override this to provide complex default behavior, for example, setting default values as a function of the values of other fields. The arguments to this method will vary depending on the type of Config.

validate_config()[source]#

Validate fields that should be checked after update is called.

This is to complement the builtin validation that Pydantic performs at the time of object construction.

validate_list(field: str, valid_options: List[str])#

Validate a list field.

Parameters
  • field (str) – name of field to validate

  • valid_options (List[str]) – values that field is allowed to take

Raises

ConfigError – if field is invalid

property all_scenes: List[SceneConfig]#