Dense-Scene-Matching

This repository contains code of our CVPR 2021 paper - "Learning Camera Localization via Dense Scene Matching" by Shitao Tang, Chengzhou Tang, Rui Huang, Siyu Zhu and Ping Tan.

This paper presents a new method for scene agnostic camera localization using dense scene matching (DSM), where a cost volume is constructed between a query image and a scene. The cost volume and the corresponding coordinates are processed by a CNN to predict dense coordinates. Camera poses can then be solved by PnP algorithms.

If you find this project useful, please cite:

@inproceedings{Tang2021Learning,
  title={Learning Camera Localization via Dense Scene Matching},
  author={Shitao Tang, Chengzhou Tang, Rui Huang, Siyu Zhu and Ping Tan},
  booktitle={Computer Vision and Pattern Recognition (CVPR)},
  year={2021}
}

Usage

Environment

  • The codes are tested along with
    • pytorch=1.4.0
    • lmdb (optional)
    • yaml
    • skimage
    • opencv
    • numpy=1.17
    • tensorboard

Installation

  • Build PyTorch operations
      cd libs/model/ops
  python setup.py install
  • Build PnP algorithm
      cd libs/utils/lm_pnp
  mkdir build
  cd build
  cmake ..
  make all

Train and Test

  • Download

    You can download the trained models and label files for 7scenes, Cambridge, Scannet.

    For 7scenes, you can use the prepared data in the following.

    Chess Fire Heads Office Pumpkin Kitchen Stairs

    For Cambridge landmarks, you can download image files here, and depths here.

  • Test

    Please refer to configs/7scenes.yaml for detailed explaination of how to set label file path and image file path.

    • 7scenes
      python tools/video_test.py --config configs/7scenes.yaml
    • Camrbrige
      python tools/video_test.py --config configs/cambridge.yaml

  • Train

    We use ResNet-FPN pretrained model.

      python tools/train_net.py

GitHub

https://github.com/Tangshitao/Dense-Scene-Matching