Examples¶

This contains examples of using Raster Vision on open datasets. Unless otherwise stated, all commands should be run inside the Raster Vision Docker container. See Docker Images for info on how to do this.

How to Run an Example¶

There is a common structure across all of the examples which represents a best practice for defining experiments. Running an example involves the following steps.

Acquire raw dataset.
(Optional) Get processed dataset which is derived from the raw dataset, either using a Jupyter notebook, or by downloading the processed dataset.
(Optional) Do an abbreviated test run of the experiment on a small subset of data locally.
Run full experiment on GPU.
Inspect output
(Optional) Make predictions on new imagery

Each of the examples has several arguments that can be set on the command line:

The input data for each experiment is divided into two directories: the raw data which is publicly distributed, and the processed data which is derived from the raw data. These two directories are set using the raw_uri and processed_uri arguments.
The output generated by the experiment is stored in the directory set by the root_uri argument.
The raw_uri, processed_uri, and root_uri can each be local or remote (on S3), and don’t need to agree on whether they are local or remote.
Experiments have a test argument which runs an abbreviated experiment for testing/debugging purposes.

In the next section, we describe in detail how to run one of the examples, SpaceNet Rio Chip Classification. For other examples, we only note example-specific details.

Chip Classification: SpaceNet Rio Buildings¶

This example performs chip classification to detect buildings on the Rio AOI of the SpaceNet dataset.

Step 1: Acquire Raw Dataset¶

The dataset is stored on AWS S3 at s3://spacenet-dataset. You will need an AWS account to access this dataset, but it will not be charged for accessing it. (To forward you AWS credentials into the container, use docker/run --aws).

Optional: to run this example with the data stored locally, first copy the data using something like the following inside the container.

aws s3 sync s3://spacenet-dataset/AOIs/AOI_1_Rio/ /opt/data/spacenet-dataset/AOIs/AOI_1_Rio/

Step 2: Run the Jupyter Notebook¶

You’ll need to do some data preprocessing, which we can do in the Jupyter notebook supplied.

docker/run --jupyter [--aws]

The --aws option is only needed if pulling data from S3. In Jupyter inside the browser, navigate to the rastervision2/examples/chip_classification/spacenet_rio_data_prep.ipynb notebook. Set the URIs in the first cell and then run the rest of the notebook. Set the processed_uri to a local or S3 URI depending on where you want to run the experiment.

Step 3: Do a test run locally¶

The experiment we want to run is in spacenet_rio.py. To run this, first get to the Docker console using:

docker/run [--aws] [--gpu] [--tensorboard]

The --aws option is only needed if running experiments on AWS or using data stored on S3. The --gpu option should only be used if running on a local GPU. The --tensorboard option should be used if running locally and you would like to view Tensorboard. The test run can be executed using something like:

export RAW_URI="s3://spacenet-dataset/"
export PROCESSED_URI="/opt/data/examples/spacenet/rio/processed-data"
export ROOT_URI="/opt/data/examples/spacenet/rio/local-output"

rastervision2 run local rastervision2.examples.chip_classification.spacenet_rio \
    -a raw_uri $RAW_URI -a processed_uri $PROCESSED_URI -a root_uri $ROOT_URI \
    -a test True --splits 2

The sample above assumes that the raw data is on S3, and the processed data and output are stored locally. The raw_uri directory is assumed to contain an AOIs/AOI_1_Rio subdirectory. This runs two parallel jobs for the chip and predict commands via --splits 2. See rastervision2 --help and rastervision2 run --help for more usage information.

Note that when running with -a test True, some crops of the test scenes are created and stored in processed_uri/crops/. All of the examples that use big image files use this trick to make the experiment run faster in test mode.

After running this, the main thing to check is that it didn’t crash, and that the visualization of training and validation chips look correct. These “debug chips” for each of the data splits can be found in $ROOT_URI/train/dataloaders/.

Step 4: Run full experiment¶

To run the full experiment on GPUs using AWS Batch, use something like the following. Note that all the URIs are on S3 since remote instances will not have access to your local file system.

export RAW_URI="s3://spacenet-dataset/"
export PROCESSED_URI="s3://mybucket/examples/spacenet/rio/processed-data"
export ROOT_URI="s3://mybucket/examples/spacenet/rio/remote-output"

rastervision run aws_batch rastervision2.examples.chip_classification.spacenet_rio \
    -a raw_uri $RAW_URI -a processed_uri $PROCESSED_URI -a root_uri $ROOT_URI \
    -a test False --splits 8

For instructions on setting up AWS Batch resources and configuring Raster Vision to use them, see Setting up AWS Batch. To monitor the training process using Tensorboard, visit <public dns>:6006 for the EC2 instance running the training job.

If you would like to run on a local GPU, replace aws_batch with local, and use local URIs. To monitor the training process using Tensorboard, visit localhost:6006, assuming you used docker/run --tensorboard.

Step 5: Inspect results¶

After everything completes, which should take about 1.5 hours if you’re running on AWS using a p3.2xlarge instance for training and 8 splits, you should be able to find the predictions over the validation scenes in $root_uri/predict/. The evaluation metrics can be found in $root_uri/eval/eval.json. This is an example of the scores from a run, which show an F1 score of 0.96 for detecting chips with buildings.

[
    {
        "gt_count": 1460.0,
        "count_error": 0.0,
        "f1": 0.962031922725018,
        "class_name": "building",
        "recall": 0.9527397260273971,
        "precision": 0.9716098420590342,
        "class_id": 1
    },
    {
        "gt_count": 2314.0,
        "count_error": 0.0,
        "f1": 0.9763865660344931,
        "class_name": "no_building",
        "recall": 0.9822817631806394,
        "precision": 0.9706292067263268,
        "class_id": 2
    },
    {
        "gt_count": 3774.0,
        "count_error": 0.0,
        "f1": 0.970833365390128,
        "class_name": "average",
        "recall": 0.9708532061473236,
        "precision": 0.9710085728062825,
        "class_id": -1
    }
]

Step 6: Predict on new imagery¶

After running an experiment, a model bundle is saved into $root_uri/bundle/. This can be used to make predictions on new images. See the Model Zoo section for more details.

Visualization using QGIS¶

To visualize a Raster Vision experiment, you can use QGIS to display the imagery, ground truth, and predictions associated with each scene. Although it’s possible to just drag and drop files into QGIS, it’s often more convenient to write a script to do this. Here is an example of a script to visualize the results for Semantic Segmentation: SpaceNet Vegas.

Semantic Segmentation: SpaceNet Vegas¶

This experiment contains an example of doing semantic segmentation using the SpaceNet Vegas dataset which has labels in vector form. It allows for training a model to predict buildings or roads. Note that for buildings, polygon output in the form of GeoJSON files will be saved to the predict directory alongside the GeoTIFF files. In addition, a vector evaluation file using SpaceNet metrics will be saved to the eval directory.

Arguments:

raw_uri should be set to the root of the SpaceNet data repository, which is at s3://spacenet-dataset, or a local copy of it. A copy only needs to contain the AOIs/AOI_2_Vegas subdirectory.
target can be buildings or roads
processed_uri should not be set because there is no processed data in this example.

Below are sample predictions and eval metrics.

Buildings¶

[
    {
        "class_id": 1,
        "precision": 0.9166443308607926,
        "recall": 0.7788752910479124,
        "gt_count": 62924777,
        "count_error": 31524.39656560088,
        "class_name": "Building",
        "f1": 0.8387483150445183
    },
    {
        "class_id": 2,
        "precision": 0.9480938442744736,
        "recall": 0.9648479452702291,
        "gt_count": 262400223,
        "count_error": 29476.379317139523,
        "class_name": "Background",
        "f1": 0.9527945047747147
    },
    {
        "class_id": null,
        "precision": 0.942010839223173,
        "recall": 0.9288768769691843,
        "gt_count": 325325000,
        "count_error": 29872.509429032507,
        "class_name": "average",
        "f1": 0.930735545099091
    }
]

Roads¶

[
    {
        "count_error": 131320.3497452814,
        "precision": 0.79827727905979,
        "f1": 0.7733719736453241,
        "class_name": "Road",
        "class_id": 1,
        "recall": 0.7574370618553649,
        "gt_count": 47364639
    },
    {
        "count_error": 213788.03361026093,
        "precision": 0.9557015578601281,
        "f1": 0.909516065847437,
        "class_name": "Background",
        "class_id": 2,
        "recall": 0.8988113906793058,
        "gt_count": 283875361
    },
    {
        "count_error": 201995.82229692052,
        "precision": 0.9331911601569118,
        "f1": 0.8900485625895702,
        "class_name": "average",
        "class_id": null,
        "recall": 0.8785960059171598,
        "gt_count": 331240000
    }
]

Semantic Segmentation: ISPRS Potsdam¶

This experiment performs semantic segmentation on the ISPRS Potsdam dataset. The dataset consists of 5cm aerial imagery over Potsdam, Germany, segmented into six classes including building, tree, low vegetation, impervious, car, and clutter. For more info see our blog post.

Data:

The dataset can only be downloaded after filling in this request form. After your request is granted, follow the link to ‘POTSDAM 2D LABELING’ and download and unzip 4_Ortho_RGBIR.zip, and 5_Labels_for_participants.zip into a directory, and then upload to S3 if desired.

Arguments:

raw_uri should contain 4_Ortho_RGBIR and 5_Labels_for_participants subdirectories.
processed_uri should be set to a directory which will be used to store test crops.

Below are sample predictions and eval metrics.

[
        {
            "precision": 0.9003686311706696,
            "recall": 0.8951149482868683,
            "f1": 0.8973353554371246,
            "count_error": 129486.40233074076,
            "gt_count": 1746655.0,
            "conf_mat": [
                0.0,
                1563457.0,
                7796.0,
                5679.0,
                10811.0,
                126943.0,
                31969.0
            ],
            "class_id": 1,
            "class_name": "Car"
        },
        {
            "precision": 0.9630047813515502,
            "recall": 0.9427071079228886,
            "f1": 0.9525027991356272,
            "count_error": 1000118.8466519706,
            "gt_count": 28166583.0,
            "conf_mat": [
                0.0,
                6976.0,
                26552838.0,
                743241.0,
                71031.0,
                556772.0,
                235725.0
            ],
            "class_id": 2,
            "class_name": "Building"
        },
        {
            "precision": 0.8466609755403327,
            "recall": 0.8983221897241067,
            "f1": 0.8715991836041085,
            "count_error": 3027173.8852443425,
            "gt_count": 30140893.0,
            "conf_mat": [
                0.0,
                4306.0,
                257258.0,
                27076233.0,
                1405095.0,
                1110647.0,
                287354.0
            ],
            "class_id": 3,
            "class_name": "Low Vegetation"
        },
        {
            "precision": 0.883517319858661,
            "recall": 0.8089167109558072,
            "f1": 0.8439042868078945,
            "count_error": 1882745.6869677808,
            "gt_count": 16928529.0,
            "conf_mat": [
                0.0,
                34522.0,
                157012.0,
                2484523.0,
                13693770.0,
                485790.0,
                72912.0
            ],
            "class_id": 4,
            "class_name": "Tree"
        },
        {
            "precision": 0.9123212945945467,
            "recall": 0.9110533473255575,
            "f1": 0.9115789047144218,
            "count_error": 1785561.1048684688,
            "gt_count": 29352493.0,
            "conf_mat": [
                0.0,
                99015.0,
                451628.0,
                1307686.0,
                262292.0,
                26741687.0,
                490185.0
            ],
            "class_id": 5,
            "class_name": "Impervious"
        },
        {
            "precision": 0.42014399072332975,
            "recall": 0.47418711749488085,
            "f1": 0.44406088467218563,
            "count_error": 787395.6814824425,
            "gt_count": 1664847.0,
            "conf_mat": [
                0.0,
                28642.0,
                157364.0,
                340012.0,
                59034.0,
                290346.0,
                789449.0
            ],
            "class_id": 6,
            "class_name": "Clutter"
        },
        {
            "precision": 0.8949197573420392,
            "recall": 0.8927540185185187,
            "f1": 0.8930493260224918,
            "count_error": 1900291.674768574,
            "gt_count": 108000000.0,
            "conf_mat": [
                [
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0,
                    0.0
                ],
                [
                    0.0,
                    1563457.0,
                    7796.0,
                    5679.0,
                    10811.0,
                    126943.0,
                    31969.0
                ],
                [
                    0.0,
                    6976.0,
                    26552838.0,
                    743241.0,
                    71031.0,
                    556772.0,
                    235725.0
                ],
                [
                    0.0,
                    4306.0,
                    257258.0,
                    27076233.0,
                    1405095.0,
                    1110647.0,
                    287354.0
                ],
                [
                    0.0,
                    34522.0,
                    157012.0,
                    2484523.0,
                    13693770.0,
                    485790.0,
                    72912.0
                ],
                [
                    0.0,
                    99015.0,
                    451628.0,
                    1307686.0,
                    262292.0,
                    26741687.0,
                    490185.0
                ],
                [
                    0.0,
                    28642.0,
                    157364.0,
                    340012.0,
                    59034.0,
                    290346.0,
                    789449.0
                ]
            ],
            "class_id": null,
            "class_name": "average"
        }
]

Object Detection: COWC Potsdam Cars¶

This experiment performs object detection on cars with the Cars Overhead With Context dataset over Potsdam, Germany.

Data:

The imagery can only be downloaded after filling in this request form. After your request is granted, follow the link to ‘POTSDAM 2D LABELING’ and download and unzip 4_Ortho_RGBIR.zip into a directory, and then upload to S3 if desired. (This step uses the same imagery as Semantic Segmentation: ISPRS Potsdam.)
Download the processed labels and unzip. These files were generated from the COWC car detection dataset using some scripts. TODO: Get these scripts into runnable shape.

Arguments:

raw_uri should point to the imagery directory created above, and should contain the 4_Ortho_RGBIR subdirectory.
processed_uri should point to the labels directory created above. It should contain the labels/all subdirectory.

Below are sample predictions and eval metrics.

[
    {
        "precision": 0.9390652367984924,
        "recall": 0.9524752475247524,
        "f1": 0.945173902480464,
        "count_error": 0.015841584158415842,
        "gt_count": 505.0,
        "class_id": 1,
        "class_name": "vehicle"
    },
    {
        "precision": 0.9390652367984924,
        "recall": 0.9524752475247524,
        "f1": 0.945173902480464,
        "count_error": 0.015841584158415842,
        "gt_count": 505.0,
        "class_id": null,
        "class_name": "average"
    }
]

Object Detection: xView Vehicles¶

This experiment performs object detection to find vehicles using the DIUx xView Detection Challenge dataset.

Data:

Sign up for an account for the DIUx xView Detection Challenge. Navigate to the downloads page and download the zipped training images and labels. Unzip both of these files and place their contents in a directory, and upload to S3 if desired.
Run the xview-data-prep.ipynb Jupyter notebook, pointing the raw_uri to the directory created above.

Arguments:

The raw_uri should point to the directory created above, and contain a labels GeoJSON file named xView_train.geojson, and a directory named train_images.
The processed_uri should point to the processed data generated by the notebook.

Below are sample predictions and eval metrics.

[
    {
        "class_name": "vehicle",
        "precision": 0.4789625193065175,
        "class_id": 1,
        "f1": 0.4036499117825103,
        "recall": 0.3597840599059615,
        "count_error": -0.2613920009287745,
        "gt_count": 17227
    },
    {
        "class_name": "average",
        "precision": 0.4789625193065175,
        "class_id": null,
        "f1": 0.4036499117825103,
        "recall": 0.3597840599059615,
        "count_error": -0.2613920009287745,
        "gt_count": 17227
    }
]

Model Zoo¶

Using the Model Zoo, you can download model bundles which contain pre-trained models and meta-data, and then run them on sample test images that the model wasn’t trained on.

rastervision2 predict <model bundle> <infile> <outfile>

Note that the input file is assumed to have the same channel order and statistics as the images the model was trained on. See rastervision predict --help to see options for manually overriding these. It shouldn’t take more than a minute on a CPU to make predictions for each sample. For some of the examples, there are also model files that can be used for fine-tuning on another dataset.

Disclaimer: These models are provided for testing and demonstration purposes and aren’t particularly accurate. As is usually the case for deep learning models, the accuracy drops greatly when used on input that is outside the training distribution. In other words, a model trained on one city probably won’t work well on another city (unless they are very similar) or at a different imagery resolution.

When unzipped, the model bundle contains a model.pth file which can be used for fine-tuning.

Model Zoo¶
Dataset	Task	Model Type	Model Bundle	Sample Image
SpaceNet Rio Buildings	Chip Classification	Resnet 50	link	link
SpaceNet Vegas Buildings	Semantic Segmentation	DeeplabV3/Resnet50	link	link
SpaceNet Vegas Roads	Semantic Segmentation	DeeplabV3/Resnet50	link	link
ISPRS Potsdam	Semantic Segmentation	DeeplabV3/Resnet50	link	link
COWC Potsdam (Cars)	Object Detection	Faster-RCNN/Resnet50	link	link

Examples¶

How to Run an Example¶

Chip Classification: SpaceNet Rio Buildings¶

Step 1: Acquire Raw Dataset¶

Step 2: Run the Jupyter Notebook¶

Step 3: Do a test run locally¶

Step 4: Run full experiment¶

Step 5: Inspect results¶

Step 6: Predict on new imagery¶

Visualization using QGIS¶

Semantic Segmentation: SpaceNet Vegas¶

Buildings¶

Roads¶

Semantic Segmentation: ISPRS Potsdam¶

Object Detection: COWC Potsdam Cars¶

Object Detection: xView Vehicles¶

Model Zoo¶

Project Links

Contents

Navigation

Versions