Geometry Processing with Neural Fields
Pytorch implementation for the NeurIPS 2021 paper:
Geometry Processing with Neural Fields
Guandao Yang,
Serge Belongie,
Bharath Hariharan,
Vladlen Koltun
Introduction
Most existing geometry processing algorithms use meshes as the default shape representation.
Manipulating meshes, however, requires one to maintain high quality in the surface discretization.
For example, changing the topology of a mesh usually requires additional procedures such as remeshing.
This paper instead proposes the use of neural fields for geometry processing.
Neural fields can compactly store complicated shapes without spatial discretization.
Moreover, neural fields are infinitely differentiable, which allows them to be optimized for objectives that involve higher-order derivatives.
This raises the question: can geometry processing be done entirely using neural fields?
We introduce loss functions and architectures to show that some of the most challenging geometry processing tasks, such as deformation and filtering, can be done with neural fields.
Experimental results show that our methods are on par with the well-established mesh-based methods without committing to a particular surface discretization.
Installation
This repository provides a Anaconda environment, and requires NVIDIA GPU to run the
optimization routine.
The code is tested in CUDA 10.2, ubuntu 16.04.
The environment can be set-up using the following commands:
conda env create -f environment.yml
conda activate NFGP
Dataset
We offer pre-processed dataset for reproducing the experimental results in the paper.
The instruciton is in section Download Dataset.
If you want to run experiment for your own shape, you need to prepare data in the following two steps:
- Create SDF samples for the shape (See section Create SDF Samples)
- Train a neural field to fit those samples (See section Create Input Neural Fields).
- Depending on the tasks, create the user specified input. For deofmration task, please refer to section Create
Deformation Handles. For sharpening and smoothing, you can directly set it through
the configuration file or the hyper-parameter.
Download Dataset
Available soon.
Create SDF Samples
Following command creates SDF ground truth samples for obtaining the Neural Fields that approximates the SDF of the
shape.
python scripts/prep_sdf_data.py <mesh_file_name> --out_path data/<mesh_file_name>
# Example
python scripts/prep_sdf_data.py armidillo.obj --out_path data/armidillo
The data preprocessing pipeline will sample 5M points uniformly withint [-1, 1]^3 and another 5M near surface (i.e
. surface plus a Gaussian with standard deviation of 0.1).
We then use the mesh_to_sdf package to compute the SDF of the points.
These points will be saved to following files:
<out_path>/mesh.obj: a copy of the original mesh.<out_path>/sdf.npy: a numpy file that contains the sampled points, their SDF, and the original mesh.
Note that this pipeline works the best with watertight mesh.
This processing pipeline takes about 5 – 10 minutes to finish for a mesh with 30k faces.
Create Input Neural Fields
Once you’ve obtained the sdf.npy file in the previous subsection,
you can use those data to train a neural field:
python train.py configs/recon/create_neural_fields.yaml --hparams data.path=<your_sdf.npy>
You can also create your own config following the examples in folder configs/recon.
Create Deformation Handles
Available soon.
Smoothing and Sharpening
To run our methods on the smoothing or sharpening task, you can use the following configurations:
# Armadillo
python train.py configs/filtering/filtering_Armadillo_beta0.yaml # smoothing
python train.py configs/filtering/filtering_Armadillo_beta2.yaml # sharpening
# Noisy sphere
python train.py configs/filtering/filtering_HalfNoisySphere_beta0.yaml # smoothing
python train.py configs/filtering/filtering_HalfNoisySphere_beta2.yaml # sharpening
# Noisy torus
python train.py configs/filtering/filtering_NoisyTorus_beta0.yaml # smoothing
python train.py configs/filtering/filtering_NoisyTorus_beta2.yaml # sharpening
You can change the value of beta by adding --hparams trainer.beta=<yourbeta>.
To run it on a different shape, you need to change the models.decoder to load the appropriate neural fields.
Deformation
Available soon.
