Merging Feature Learning and Camera Pose Estimation

Neural Reprojection Error

Neural Reprojection Error: Merging Feature Learning and Camera Pose Estimation (CVPR 2021)
This is the official repository for our paper, which contains model weights and source code to perform NRE-based absolute camera pose estimation.

Dependencies

This repository was tested with the following packages, other versions should be compatible:

Python 3.8.5
PyTorch 1.8.1
PyTorch Lightning 1.2.10
OpenCV 4.5.1
Plotly 4.10.0
NumPy 1.19.2
coloredlogs 15.0

They can all be installed following command:
pip3 install numpy opencv-python torch torchvision pytorch-lightning plotly coloredlogs

Running

For a given pair of images, localizing with the NRE-based pose estimator is as simple as:

carbon

Try it

You can run NRE-based camera pose estimation on a Megadepth validation sample in example.ipynb.
You will need to have the aforementionned dependencies installed, as well as a GPU with 8Gb+ memory.
For simple visualization of the notebook try this link.

Additional parameters

For additional control over NRE-based camera pose estimation, you can tweak several parameters in config.yml.
Notably:

skip_gnc: Skip the coarse and fine GNC-IRLS refinement steps
n_iter_p3p: The number of P3P iterations
top_n_p3p: The fraction of top-N maps to run P3P.
max_fine_imsize: The maximum image size for the fine forward pass. This can be reduced if you encounter memory issues, although this will most likely lead to loss in accuracy.
coarse_sigma_max and coarse_sigma_min: Control the sigma upper and lower boundaries for the coarse GNC-IRLS. Analogous parameters are available for the fine GNC-IRLS.
use_re=True: This additional parameter can be passed in the estimator.localize() method. This setting corresponds to Eq. 11 minimization in the paper).

BibTex

Please consider citing our work:

@InProceedings{Germain_2021_CVPR,
    author    = {Germain, Hugo and Lepetit, Vincent and Bourmaud, Guillaume},
    title     = {Neural Reprojection Error: Merging Feature Learning and Camera Pose Estimation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2021},
    pages     = {414-423}
}