Aerial Depth Completion
This work is described in the letter "Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation", by Lucas Teixeira, Martin R. Oswald, Marc Pollefeys, Margarita Chli, published in the IEEE Robotics and Automation Letters (RA-L / ICRA) ETHZ Library link.
Video:
Presentation:
Citations:
If you use this Code or Aerial Dataset, please cite the following publication:
@article{Teixeira:etal:RAL2020,
title = {{Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation}},
author = {Lucas Teixeira and Martin R. Oswald and Marc Pollefeys and Margarita Chli},
journal = {{IEEE} Robotics and Automation Letters ({RA-L})},
doi = {10.1109/LRA.2020.2967296},
year = {2020}
}
NYUv2, CAB and PVS datasets require further citation from their authors.
During our research, we reformat and created ground-truth depth for the CAB and PVS datasets.
This code also contains thirt-party networks used for comparison.
Please also cite their authors properly in case of use.
Acknowledgment:
The authors thank Fangchang Ma and Abdelrahman Eldesokey for sharing their code that is partially used here. The authors also thanks the owner of the 3D models used to build the dataset. They are identified in each 3D model file.
Data and Simulator
Trained Models
Several trained models are available - here.
Datasets
- Aerial Dataset - link
- NYUv2 Dataset - link (preprocessed by Fangchang Ma and originally from Silberman et al. ECCV12)
- CAB Dataset - on request (In this work, we created the depth information for the dataset originally published in Teixeira and Chli IROS16)
- PVS Dataset - on request (In this work, we created the depth information for the dataset originally published in Restrepo et al. P&RS14)
To be used together by our code, the datasets need to be merged, this means that the content of the train folder of each dataset need to be place in a single train folder. The same happens with the eval folder.
Simulator
The Aerial Dataset was created using this simulator link.
3D Models
Most of the 3D models used to create the dataset can be download here. In the license files contain the authors of the 3D models. Some models were extended with a satellite image from Google Earth.
Running the code
Prerequisites
- PyTorch 1.0.1
- Python 3.6
- Plus dependencies
Testing Example
python3 main.py --evaluate "/media/lucas/lucas-ds2-1tb/tmp/model_best.pth.tar" --data-path "/media/lucas/lucas-ds2-1tb/dataset_big_v12"
Training Example
python3 main.py --data-path "/media/lucas/lucas-ds2-1tb/dataset_big_v12" --workers 8 -lr 0.00001 --batch-size 1 --dcnet-arch gudepthcompnet18 --training-mode dc1_only --criterion l2
python3 main.py --data-path "/media/lucas/lucas-ds2-1tb/dataset_big_v12" --workers 8 --criterion l2 --training-mode dc0-cf1-ln1 --dcnet-arch ged_depthcompnet --dcnet-pretrained /media/lucas/lucas-ds2-1tb/tmp/model_best.pth.tar:dc_weights --confnet-arch cbr3-c1 --confnet-pretrained /media/lucas/lucas-ds2-1tb/tmp/model_best.pth.tar:conf_weights --lossnet-arch ged_depthcompnet --lossnet-pretrained /media/lucas/lucas-ds2-1tb/tmp/model_best.pth.tar:lossdc_weights
Parameters
| Parameter | Description |
|---|---|
| --help | show this help message and exit |
| --output NAME | output base name in the subfolder results |
| --training-mode ARCH | this variable indicating the training mode. Our framework has up to tree parts the dc (depth completion net), the cf (confidence estimation net) and the ln (loss net). The number 0 or 1 indicates whether the network should be updated during the back-propagation. All the networks can be pre-load using other parameters. training_mode: dc1_only ; dc1-ln0 ; dc1-ln1 ; dc0-cf1-ln0 ; dc1-cf1-ln0 ; dc0-cf1-ln1 ; dc1-cf1-ln1 (default: dc1_only) |
| --dcnet-arch ARCH | model architecture: resnet18 ; udepthcompnet18 ; gms_depthcompnet ; ged_depthcompnet ; gudepthcompnet18 (default: resnet18) |
| --dcnet-pretrained PATH | path to pretraining checkpoint for the dc net (default: empty). Each checkpoint can have multiple network. So it is necessary to define each one. the format is path:network_name. network_name can be: dc_weights, conf_weights, lossdc_weights. |
| --dcnet-modality MODALITY | modality: rgb ; rgbd ; rgbdw (default: rgbd) |
| --confnet-arch ARCH | model architecture: cbr3-c1 ; cbr3-cbr1-c1 ; cbr3-cbr1-c1res ; join ; none (default: cbr3-c1) |
| --confnet-pretrained PATH | path to pretraining checkpoint for the cf net (default: empty). Each checkpoint can have multiple network. So it is necessary to define each one. the format is path:network_name. network_name can be: dc_weights, conf_weights, lossdc_weights. |
| --lossnet-arch ARCH | model architecture: resnet18 ; udepthcompnet18 (uresnet18) ; gms_depthcompnet (nconv-ms) ; ged_depthcompnet (nconv-ed) ; gudepthcompnet18 (nconv-uresnet18) (default: ged_depthcompnet) |
| --lossnet-pretrained PATH | path to pretraining checkpoint for the ln net (default: empty). Each checkpoint can have multiple network. So it is necessary to define each one. the format is path:network_name. network_name can be: dc_weights, conf_weights, lossdc_weights. |
| --data-type DATA | dataset: visim ; kitti (default: visim) |
| --data-path PATH | path to data folder - this folder has to have inside a val folder and a train folder if it is not in evaluation mode. |
| --data-modality MODALITY | this field define the input modality in the format colour-depth-weight. kfd and fd mean random sampling in the ground-truth. kgt means keypoints from slam with depth from ground-truth. kor means keypoints from SLAM with depth from the landmark. The weight can be binary (bin) or from the uncertanty from slam (kw). The parameter can be one of the following: rgb-fd-bin ; rgb-kfd-bin ; rgb-kgt-bin ; rgb-kor-bin ; rgb-kor-kw (default: rgb-fd-bin) |
| --workers N | number of data loading workers (default: 10) |
| --epochs N | number of total epochs to run (default: 15) |
| --max-gt-depth D | cut-off depth of ground truth, negative values means infinity (default: inf [m]) |
| --min-depth D | cut-off depth of sparsifier (default: 0 [m]) |
| --max-depth D | cut-off depth of sparsifier, negative values means infinity (default: inf [m]) |
| --divider D | Normalization factor - zero means per frame (default: 0 [m]) |
| --num-samples N | number of sparse depth samples (default: 500) |
| --sparsifier SPARSIFIER | sparsifier: uar ; sim_stereo (default: uar) |
| --criterion LOSS | loss function: l1 ; l2 ; il1 (inverted L1) ; absrel (default: l1) |
| --optimizer OPTIMIZER | Optimizer: sgd ; adam (default: adam) |
| --batch-size BATCH_SIZE | mini-batch size (default: 8) |
| --learning-rate LR | initial learning rate (default 0.001) |
| --learning-rate-step LRS | number of epochs between reduce the learning rate by 10 (default: 5) |
| --learning-rate-multiplicator LRM | multiplicator (default 0.1) |
| --momentum M | momentum (default: 0) |
| --weight-decay W | weight decay (default: 0) |
| --val-images N | number of images in the validation image (default: 10) |
| --print-freq N | print frequency (default: 10) |
| --resume PATH | path to latest checkpoint (default: empty) |
| --evaluate PATH | evaluates the model on validation set, all the training parameters will be ignored, but the input parameters still matters (default: empty) |
| --precision-recall | enables the calculation of precision recall table, might be necessary to ajust the bin and top values in the ConfidencePixelwiseThrAverageMeter class. The result table shows for each confidence threshold the error and the density (default:false) |
| --confidence-threshold VALUE | confidence threshold , the best way to select this number is create the precision-recall table. (default: 0) |
Contact
In case of any issue, fell free to contact me via email lteixeira at mavt.ethz.ch.
