Mirrored-Human

Code for "Reconstructing 3D Human Pose by Watching Humans in the Mirror", CVPR 2021 oral.

smpl-avatar

calorie

imitate

Features

In this paper, we introduce the new task of reconstructing 3D human pose from a single image in which we can see the person and the person’s image through a mirror.

This implementation:

  • has the demo of our optimization-based approach implemented purely in PyTorch.
  • provides a method to estimate the surface normal of the mirror from vanishing points.
  • provides an annotator to label the mirror edges for the vanishing points.
  • can estimate the focal length of the Internet mirror images.

Installation

This repo has a close relation with EasyMocap. Please refer to our EasyMocap project for installation.

Demo

Download our zju-m-test.zip and run the following code:

# set the data path
data=<path_to_sample>/zju-m-demo
out=<path_to_sample>/zju-m-demo-output
# extract the video frames
python3 scripts/preprocess/extract_video.py ${data}
# Run demo on videos
python3 apps/demo/1v1p_mirror.py ${data} --out ${out} --vis_smpl --video

Mirrored-Human Dataset (Coming Soon)

Due to the license limitation, we cannot share the raw data directly. We are working hard to organize the Mirrored-Human dataset in terms of url links and timestamps.

See Build Your Internet Dataset if you can't wait for our release.

dataset_show

Annotator

We also provide the annotator metioned in our paper.

The first row shows that we label the edges of the mirror and calculate the vanishing point by the human body automaticly. The intrisic camera parameter can be calculated by this two vanishing points.

The second row shows that to obtain a more accurate vanishing points and camera parameters, we can label the parallel lines in the scene, for example the door, the grid in the ground, and the door.

vanish_body

vanish_body0

vanish_scene

vanish_scene0

See EasyMocap/apps/annotator for more instructions.

Build Custom Internet Dataset

See doc/internet.md for more instructions.

Build Custom Evaluation Dataset (Multi-View)

This part is provided for the researchers who want to:

  1. capture more accurate human motion with multiple cameras and a mirror
  2. build a different evaluation dataset

See doc/custom.md for more instructions.

000530_mesh

Evaluation

To evaluate the reconstruction part in our paper, see doc/evaluation.md.

Contact

Please open an issue if you have any questions. We appreciate all contributions to improve our project.

If you find some videos that we didn't notice, please tell us.

Citation

@inproceedings{fang2021mirrored,
  title={Reconstructing 3D Human Pose by Watching Humans in the Mirror},
  author={Fang, Qi and Shuai, Qing and Dong, Junting and Bao, Hujun and Zhou, Xiaowei},
  booktitle={CVPR},
  year={2021}
}

Acknowledgement

This project is build on our EasyMocap. We also would like to thank Jianan Zhen and Yuhao Chen for their advice for the paper. Sincere thanks to the performers (Yuji Chen and Hao Xu) in the evaluation dataset and people who uploaded the mirror-human videos to the Internet.

Qi Fang*, Qing Shuai*, Junting Dong, Hujun Bao, Xiaowei Zhou
CVPR 2021 Oral

GitHub

https://github.com/zju3dv/Mirrored-Human