Neural Deformation Graphs
Neural Deformation Graphs for Globallyconsistent Nonrigid Reconstruction
Aljaž Božič, Pablo Palafox, Michael Zollhöfer, Justus Thies, Angela Dai, Matthias Nießner
CVPR 2021 (Oral Presentation)
This repository contains the code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globallyconsistent deformation tracking and 3D reconstruction of nonrigid objects.
Specifically, we implicitly model a deformation graph via a deep neural network and empose perframe viewpoint consistency as well as interframe graph and surface consistency constraints in a selfsupervised fashion.
That results in a differentiable construction of a deformation graph that is able to handle deformations present in the whole sequence.
Install all dependencies

Download the latest conda here.

To create a conda environment with all the required packages using conda run the following command:
conda env create f resources/env.yml
The above command creates a conda environment with the name ndg.

Compile external dependencies inside
external
directory by executing:conda activate ndg
./build_external.sh
The external dependencies are PyMarchingCubes, gaps and Eigen.
Generate data for visualization & training
In our experiments we use depth inputs from 4 camera views. These depth maps were captured with 4 Kinect Azure sensors. For quantitative evaluation we also used synthetic data, where 4 depth views were rendered from ground truth meshes. In both cases, screened Poisson reconstruction (implemented in MeshLab) was used to obtain meshes for data generation. An example sequence of meshes of a synthetic doozy sequence can be downloaded here.
To generate training data from these meshes, they need to be put into a directory out/meshes/doozy
. Then the following code executes data generation, producing generated data samples in out/dataset/doozy
:
./generate_data.sh
Visualize neural deformation graphs using pretrained models
After data generation you can already check out the neural deformation graph estimation using a pretrained model checkpoint. You need to place it into the out/models
directory, and run visualization:
./viz.sh
Reconstruction visualization can take longer, if you want to check out graphs only, you can uncomment viz_only_graph
argument in viz.sh
.
Within the Open3D viewer, you can navigate different settings using these keys:
N
: toggle graph nodes and edgesG
: toggle ground truthD
: show nextA
: show previousS
: toggle smooth shading
Train a model from scratch
You can train a model from scratch using train_graph.sh
and train_shape.sh
scripts, in that order. The model checkpoints and tensorboard stats are going to be stored into out/experiments
.
Optimize graph
To estimate a neural deformation graph from input observations, you need to specify the dataset to be used (inside out/dataset
, should be generated before hand), and then training can be started using the following script:
./train_graph.sh
We ran all our experiments on NVidia 2080Ti GPU, for about 500k iterations. After the model has converged, you can visualize the optimized neural deformation graph using viz.sh
script.
To check out convergence, you can visualize loss curves with tensorboard
by running the following inside out/experiments
directory:
tensorboard logdir=.
Optimize shape
To optimize shape, you need to initialize the graph with a pretrained graph model. That means that inside train_shape.sh
you need to specify the graph_model_path
, which should point to the converged checkpoint of the graph model (graph model usually converges at around 500k iterations). MultiMLP model can then be optimized to reconstruct shape geometry by running:
./train_shape.sh
Similar to graph optimization also shape optimization converges in about 500k iterations.
Citation
If you find our work useful in your research, please consider citing:
@article{bozic2021neuraldeformationgraphs,
title={Neural Deformation Graphs for Globallyconsistent Nonrigid Reconstruction},
author={Bo{\v{z}}i{\v{c}}, Alja{\v{z}} and Palafox, Pablo and Zollh{\"o}fer, Michael and Dai, Angela and Thies, Justus and Nie{\ss}ner, Matthias},
journal={CVPR},
year={2021}
}