PETR: Position Embedding Transformation for Multi-View 3D Object Detection

This repository is an official implementation of PETR: Position Embedding Transformation for Multi-View 3D Object Detection.

In this paper, we develop position embedding transformation (PETR) for multi-view 3D object detection. PETR encodes the position information of 3D coordinates into image features, producing the 3D position-aware features. Object query can perceive the 3D position-aware features and perform end-to-end object detection. It can serve as a simple yet strong baseline for future research.

News

2022.06.10 The code of PETR is released. 2022.06.06 PETRv2 is released on arxiv. 2022.06.01 PETRv2 achieves another SOTA performance on nuScenes dataset (58.2% NDS and 49.0% mAP) by the temporal modeling and supports BEV segmentation. 2022.03.10 PETR is released on arxiv. 2022.03.08 PETR achieves SOTA performance (50.4% NDS and 44.1% mAP) on standard nuScenes dataset.

Preparation

This implementation is built upon detr3d, which can be constructed as the instructions.

Environments Linux, Python==3.6.8, CUDA == 11.2, pytorch == 1.9.0, mmdet3d == 0.17.1
```
pip install -r requirements.txt
```
Data Follow the mmdet3d to process the nuScenes dataset (https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/data_preparation.md).
Pretrained weights To verify the performance on the val set, we provide the pretrained V2-99 weights. The V2-99 is pretrained on DDAD15M (weights) and further trained on nuScenes train set with FCOS3D. For the results on test set in the paper, we use the DD3D pretrained weights. Please put the pretrained weights into ./ckpts/.

After preparation, you will be able to see the following directory structure:

PETR
├── mmdetection3d
├── projects
│   ├── configs
│   ├── mmdet3d_plugin
├── tools
├── data
│   ├── nuscenes
├── ckpts
├── README.md

Train & inference

git clone https://github.com/megvii-research/PETR.git

cd PETR

You can train the model following:

tools/dist_train.sh projects/configs/petr/petr_r50dcn_gridmask_p4.py 8 --work-dir work_dirs/petr_r50dcn_gridmask_p4/

You can evaluate the model following:

tools/dist_test.sh projects/configs/petr/petr_r50dcn_gridmask_p4.py work_dirs/petr_r50dcn_gridmask_p4/latest.pth 8 --eval bbox

Main Results

We provide some results on nuScenes val set with pretrained models. These model are trained on 8x 2080ti without cbgs. Note that the models and logs are also available at Baidu Netdisk with code petr.

config	mAP	NDS	training	config	download
PETR-r50-c5-1408×512	30.5%	35.0%	18hours	config	log / gdrive
PETR-r50-p4-1408×512	31.70%	36.7%	21hours	config	log / gdrive
PETR-vov-p4-800×320	37.8%	42.6%	17hours	config	log / gdrive
PETR-vov-p4-1600×640	40.40%	45.5%	36hours	config	log / gdrive

Acknowledgement

Many thanks to the authors of mmdetection3d and detr3d .

Citation

If you find this project useful for your research, please consider citing:

@article{liu2022petr,
  title={Petr: Position embedding transformation for multi-view 3d object detection},
  author={Liu, Yingfei and Wang, Tiancai and Zhang, Xiangyu and Sun, Jian},
  journal={arXiv preprint arXiv:2203.05625},
  year={2022}
}

@article{liu2022petrv2,
  title={PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images},
  author={Liu, Yingfei and Yan, Junjie and Jia, Fan and Li, Shuailin and Gao, Qi and Wang, Tiancai and Zhang, Xiangyu and Sun, Jian},
  journal={arXiv preprint arXiv:2206.01256},
  year={2022}
}