UMR
Self-supervised Single-view 3D Reconstruction.
Citation
Please cite our paper if you find this code useful for your research.
@inproceedings{umr2020,
title={Self-supervised Single-view 3D Reconstruction via Semantic Consistency},
author={Li, Xueting and Liu, Sifei and Kim, Kihwan and De Mello, Shalini and Jampani, Varun and Yang, Ming-Hsuan and Kautz, Jan},
booktitle={ECCV},
year={2020}
}
Prerequisites
-
Download code & pre-trained model:
Git clone the code by:
git clone https://github.com/NVlabs/UMR $ROOTPATH
The pre-trained weights and semantic template can be downloaded from Google drive and unzip at
$ROOTPATH/UMR/
. -
Install packages:
Virtual environments are not required but highly recommended:
conda create -n umr python=3.6 source activate umr
Then run:
cd $ROOTPATH/UMR sh install.sh
Run the demo
Run the following command from the $ROOTPATH
directory:
python -m UMR.experiments.demo --model_path UMR/cachedir/snapshots/cub_net/pred_net_latest.pth --img_path UMR/demo_imgs/birdie.jpg --out_path UMR/cachedir/demo
The results will be saved at out_path
in the above command.
Quantitative Evaluation
Download CUB bird images by:
wget http://www.vision.caltech.edu/visipedia-data/CUB-200-2011/CUB_200_2011.tgz && tar -xf CUB_200_2011.tgz
Command for keypoint transfer evaluation using predicted texture flow:
python -m UMR.experiments.test_kp --model_path UMR/cachedir/snapshots/cub_net/pred_net_latest.pth --split test --number_pairs 10000 --cub_cache_dir UMR/cachedir/cub/ --cub_dir CUB_200_2011/
Command for keypoint transfer evaluation using predicted camera pose:
python -m UMR.experiments.test_kp --model_path UMR/cachedir/snapshots/cub_net/pred_net_latest.pth --split test --number_pairs 10000 --mode cam --cub_cache_dir UMR/cachedir/cub/ --cub_dir CUB_200_2011/
For keypoint transfer result visualization, add --visualize
. The visualizations will be saved at $ROOTPATH/UMR/cachedir/results_vis/
Command for evaluating mask IoU:
python -m UMR.experiments.test_iou --model_path UMR/cachedir/snapshots/cub_net/pred_net_latest.pth --split test --cub_cache_dir UMR/cachedir/cub/ --cub_dir CUB_200_2011/ --batch_size 32
Training
To train the full model with semantic consistency constraint:
-
Prepare SCOPS predictions following instructions here.
-
Download semantic template as described above if haven't done so.
-
Run the following command from the
$ROOTPATH
folder:python -m UMR.experiments.train_s2 --name=cub_s2 --batch_size 16 --cub_dir CUB_200_2011/ --cub_cache_dir UMR/cachedir/cub --scops_path SCOPS/results/cub/ITER_60000/train/dcrf_prob/ --stemp_path UMR/cachedir/cub/scops/
If the code is ran on multiple GPUs, add
--multi_gpu --gpu_num GPU_NUMBER
to the above commands.
If you wish to train from scratch (i.e. learn the semantic template from scratch):
-
Run the following command from the
$ROOTPATH
folder:python -m UMR.experiments.train_s1 --name=cub_s1 --gpu_num 4 --multi_gpu -cub_dir CUB_200_2011/ --cub_cache_dir UMR/cachedir/cub
-
Then compute the semantic template by:
python -m UMR.experiments.avg_uv --model_path UMR/cachedir/snapshots/cub_s1/pred_net_latest.pth --batch_size 16 --out_dir UMR/cachedir/snapshots/cub_s1 --cub_dir CUB_200_2011/ --cub_cache_dir UMR/cachedir/cub --scops_path SCOPS/results/cub/ITER_60000/train/dcrf_prob/
Now you can use the computed semantic template in the model training by setting
stemp_path
intrain_s2.py
toout_dir
in the above command.