This is the PyTorch implementation for inference and training of the future prediction bird's-eye view network as described in:

FIERY: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras

Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, ‪Vijay Badrinarayanan, Roberto Cipolla and Alex Kendall


Multimodal future predictions by our bird’s-eye view network.

Top two rows: RGB camera inputs. The predicted future trajectories and segmentations are projected to the ground plane in the images.

Bottom row: future instance prediction in bird’s-eye view in a 100m×100m capture size around the ego-vehicle, which is indicated by a black rectangle in the center.

If you find our work useful, please consider citing:

  title     = {{FIERY}: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras},
  author    = {Anthony Hu and Zak Murez and Nikhil Mohan and Sofía Dudas and 
               Jeffrey Hawke and Vijay Badrinarayanan and Roberto Cipolla and Alex Kendall},
  booktitle = {arXiv preprint},
  year = {2021}

⚙ Setup

  • Create the conda environment by running conda env create.

? Prediction


In a colab notebook:
Open In Colab

Or locally:

  • Download pre-trained weights.
  • Run python --checkpoint ${CHECKPOINT_PATH}. This will render predictions from the network and save
    them to an output_vis folder.


? Pre-trained models

All the configs are in the folder fiery/configs

Config Dataset Past context Future horizon BEV size IoU VPQ
baseline.yml NuScenes 1.0s 2.0s 100mx100m (50cm res.) 36.7 29.9
lyft/baseline.yml Lyft 0.8s 2.0s 100mx100m (50cm res.) 36.3 29.2
literature/pon_setting.yml NuScenes 1.0s 0.0s 100mx50m (25cm res.) 40.0 -
literature/lift_splat_setting.yml NuScenes 1.0s 0.0s 100mx100m (50cm res.) 37.8 -
literature/fishing_setting.yml NuScenes 1.0s 2.0s 32.0mx19.2m (10cm res.) 59.4 -

? Training

To train the model from scratch on NuScenes:

  • Run python --config fiery/configs/baseline.yml DATASET.DATAROOT ${NUSCENES_DATAROOT}.

To train on single GPU add the flag GPUS [0], and to change the batch size use the flag BATCHSIZE ${DESIRED_BATCHSIZE}.