Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions

Python Linux PRs Welcome Paper MIT license

example Our moving object segmentation on the unseen SemanticKITTI test sequences 18 and 21. Red points are predicted as moving.

Table of Contents

  1. Publication
  2. Overview
  3. Data
  4. Installation
  5. Running the Code
  6. Evaluation and Visualization
  7. Pretrained Model
  8. License

Publication

If you use our code in your academic work, please cite the corresponding paper:

@article{mersch2022ral,
author = {B. Mersch and X. Chen and I. Vizzo and L. Nunes and J. Behley and C. Stachniss},
title = {{Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions}},
journal={IEEE Robotics and Automation Letters (RA-L)},
year = 2022,
}

Overview

Given a sequence of point clouds, our method segments moving (red) from non-moving (black) points.

We first create a sparse 4D point cloud of all points in a given receding window. We use sparse 4D convolutions from the MinkowskiEngine to extract spatio-temporal features and predict per-points moving object scores.

Data

Download the SemanticKITTI data from the official website.

./
└── sequences
  ├── 00/           
  │   ├── velodyne/	
  |   |	├── 000000.bin
  |   |	├── 000001.bin
  |   |	└── ...
  │   └── labels/ 
  |       ├── 000000.label
  |       ├── 000001.label
  |       └── ...
  ├── 01/ # 00-10 for training
  ├── 08/ # for validation
  ├── 11/ # 11-21 for testing
  └── ...

Installation

Clone this repository in your workspace with

git clone https://github.com/PRBonn/4DMOS

With Docker

We provide a Dockerfile and a docker-compose.yaml to run all docker commands with a simple Makefile.

To use it, you need to

  1. Install Docker

  2. Install docker-compose with

    sudo apt-get install docker-compose
  3. Install the NVIDIA Container Toolkit

  4. IMPORTANT To have GPU access during the build stage, make nvidia the default runtime in /etc/docker/daemon.json:

    {
        "runtimes": {
            "nvidia": {
                "path": "/usr/bin/nvidia-container-runtime",
                "runtimeArgs": []
            } 
        },
        "default-runtime": "nvidia" 
    }

    Save the file and run sudo systemctl restart docker to restart docker.

  5. Build the image with all dependendencies with

    make build

Before running the container, you need to set the path to your dataset:

export DATA=path/to/dataset/sequences

To test that your container is running propoerly, do

make test

Finally, run the container with

make run

You can now work inside the container and run the training and inference scripts.

Without Docker

Without Docker, you need to install the dependencies specified in the setup.py. This can be done in editable mode by running

git clone [email protected]:benemer/4DMOS.git
cd 4DMOS
python3 -m pip install --editable .

When installing the MinkowskiEngine, your CUDA version has to match the CUDA version that was used to compile PyTorch. If you run into problems installing the MinkowskiEngine, please have a look at the original MinkowskiEngine repo and their installation wiki page.

Running the Code

If not done yet, specify the path to the SemanticKITTI data:

export DATA=path/to/dataset/sequences

If you use Docker, you now need to run the container with make run.

Training

To train a model with the parameters specified in config/config.yaml, run

python scripts/train.py

Find more options like loading weights from a pre-trained model or checkpointing by passing the --help flag to the command above.

Inference

Inference is done in two steps. First, predicting moving object confidence scores and second, fusing multiple confidence values to get a final prediction (non-overlapping strategy or binary Bayes filter.

To infer the per-point confidence scores for a model checkpoint at path/to/model.ckpt, run

python scripts/predict_confidences.py -w path/to/model.ckpt

We provide several additional options, see --help flag. The confidence scores are stored in predictions/ID/POSES/confidences to distinguish setups using different model IDs and pose files.

Next, the final moving object predictions can be obtained by

python scripts/confidences_to_labels.py -p predictions/ID/POSES

You can use the --strategy argument to decide between the non-overlapping or bayesian filter strategy from the paper. Run with --help to see more options. The final predictions are stored in predictions/ID/POSES/labels/.

Evaluation and Visualization

We use the SemanticKITTI API to evaluate the intersection-over-union (IOU) of the moving class as well as to visualize the predictions.

Pretrained Models

License

This project is free software made available under the MIT License. For details see the LICENSE file.

GitHub

View Github