RetrievalFuse

Paper | Project Page | Video

RetrievalFuse: Neural 3D Scene Reconstruction with a Database
Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, Angela Dai
ICCV2021

This repository contains the code for the ICCV 2021 paper RetrievalFuse, a novel approach for 3D reconstruction from low resolution distance field grids and from point clouds.

In contrast to traditional generative learned models which encode the full generative process into a neural network and can struggle with maintaining local details at the scene level, we introduce a new method that directly leverages scene geometry from the training database.

File and Folders


Broad code structure is as follows:

File / Folder Description
config/super_resolution Super-resolution experiment configs
config/surface_reconstruction Surface reconstruction experiment configs
config/base Defaults for configurations
config/config_handler.py Config file parser
data/splits Training and validation splits for different datasets
dataset/scene.py SceneHandler class for managing access to scene data samples
dataset/patched_scene_dataset.py Pytorch dataset class for scene data
external/ChamferDistancePytorch For calculating rough chamfer distance between prediction and target while training
model/attention.py Attention, folding and unfolding modules
model/loss.py Loss functions
model/refinement.py Refinement network
model/retrieval.py Retrieval network
model/unet.py U-Net model used as a backbone in refinement network
runs/ Checkpoint and visualizations for experiments dumped here
trainer/train_retrieval.py Lightning module for training retrieval network
trainer/train_refinement.py Lightning module for training refinement network
util/arguments.py Argument parsing (additional arguments apart from those in config)
util/filesystem_logger.py For copying source code for each run in the experiment log directory
util/metrics.py Rough metrics for logging during training
util/mesh_metrics.py Final metrics on meshes
util/retrieval.py Script to dump retrievals once retrieval networks have been trained; needed for training refinement.
util/visualizations.py Utility scripts for visualizations

Further, the data/ directory has the following layout

<div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="data # root data directory
├── sdf_008 # low-res (8^3) distance fields
├──
├──
├──
├──

├──

├── sdf_016 # low-res (16^3) distance fields
├──
├──
├──
├──

├──

├── sdf_064 # high-res (64^3) distance fields
├──
├──
├──
├──

├──

├── pc_20K # point cloud inputs
├──
├──
├──
├──

├──

├── splits # train/val splits
├── size # data needed by SceneHandler class (autocreated on first run)
├── occupancy # data needed by SceneHandler class (autocreated on first run)
“>

data                    # root data directory
├── sdf_008             # low-res (8^3) distance fields
    ├── 
   
         
        ├── 
    
     
        ├── 
     
      
        ├── 
      
       
        ...
    ├── 
       
         ... ├── sdf_016 # low-res (16^3) distance fields ├── 
        
          ├── 
         
           ├── 
          
            ├── 
           
             ... ├── 
            
              ... ├── sdf_064 # high-res (64^3) distance fields ├── 
             
               ├── 
              
                ├── 
               
                 ├── 
                
                  ... ├── 
                 
                   ... ├── pc_20K # point cloud inputs ├── 
                  
                    ├── 
                   
                     ├── 
                    
                      ├── 
                     
                       ... ├── 
                      
                        ... ├── splits # train/val splits ├── size # data needed by SceneHandler class (autocreated on first run) ├── occupancy # data needed by SceneHandler class (autocreated on first run)