PDC-Net has multiple inference alternative options. if model is PDC-Net, add options:

• –confidence_map_R, for computation of the confidence map p_r, default is 1.0
• –multi_stage_type in
  • ‘direct’ (D)
  • ‘homography_from_quarter_resolution_uncertainty’ (H)
  • ‘multiscale_homo_from_quarter_resolution_uncertainty’ (MS)
• –ransac_thresh, used for homography and multiscale multi-stages type, default is 1.0
• –mask_type, for thresholding the estimated confidence map and using the confident matches for internal homography estimation, for homography and multiscale multi-stage types, default is proba_interval_1_above_5
• –homography_visibility_mask, default is True
• –scaling_factors’, used for multi-scale, default are [0.5, 0.6, 0.88, 1, 1.33, 1.66, 2]

Use direct (D) when image pairs only show limited view-point changes (for example consecutive images of a video, like in the optical flow task). For larger view-point changes, use homography (H) or multi-scale (MS).

For example, to run PDC-Net with multi-scale, add at the end of the command

PDCNet --multi_stage_type multiscale_homo_from_quarter_resolution_uncertainty --mask_type proba_interval_1_above_10

PDC-Net has multiple inference alternative options. if model if PDC-Net, add options:

• –confidence_map_R, for computation of the confidence map p_r, default is 1.0
• –multi_stage_type in
  • ‘direct’ (D)
  • ‘homography_from_quarter_resolution_uncertainty’ (H)
  • ‘multiscale_homo_from_quarter_resolution_uncertainty’ (MS)
• –ransac_thresh, used for homography and multiscale multi-stages type, default is 1.0
• –mask_type, for thresholding the estimated confidence map and using the confident matches for internal homography estimation, for homography and multiscale multi-stage types, default is proba_interval_1_above_5
• –homography_visibility_mask, default is True
• –scaling_factors’, used for multi-scale, default are [0.5, 0.6, 0.88, 1, 1.33, 1.66, 2]

For example, to run PDC-Net with multi-scale, add at the end of the command

PDCNet --multi_stage_type multiscale_homo_from_quarter_resolution_uncertainty --mask_type proba_interval_1_above_10

Results using PDC-Net with multi-stage (homography_from_quarter_resolution_uncertainty, H) or multi-scale (multiscale_homo_from_quarter_resolution_uncertainty, MS) employ RANSAC internally. Therefore results may vary a bit but should remain within 1-2 %. For pose estimation, we also compute the pose with RANSAC, which leads to some variability in the results.

Data preparation: We use the test set provided in RANSAC-Flow. It is composed of 1600 pairs and also includes a csv file (‘test1600Pairs.csv’) containing the name of image pairs to evaluate and the corresponding ground-truth correspondences. Download everything with

bash assets/download_megadepth_test.sh

The resulting file structure is the following

megadepth_test_set/
└── MegaDepth/
    └── Test/
        └── test1600Pairs/  
        └── test1600Pairs.csv

Evaluation: After updating the path of ‘megadepth’ and ‘megadepth_csv’ in admin/local.py, evaluation is run with

python eval_matching.py --datasets megadepth --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir PDCNet --multi_stage_type multiscale_homo_from_quarter_resolution_uncertainty

Similar results should be obtained:

Data preparation: Images can be downloaded from the Visual Localization Challenge (at the bottom of the site), or more precisely here. The CSV file with the ground-truth correspondences can be downloaded from here. The file structure should be the following:

RobotCar
├── img/
└── test6511.csv

Evaluation: After updating the path of ‘robotcar’ and ‘robotcar_csv’ in admin/local.py, evaluation is run with

python eval_matching.py --datasets robotcar --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir PDCNet --multi_stage_type multiscale_homo_from_quarter_resolution_uncertainty

Similar results should be obtained:

Data preparation: execute ‘bash assets/download_ETH3D.sh’ from our GLU-Net repo. It does the following:

• Create your root directory ETH3D/, create two sub-directories multiview_testing/ and multiview_training/
• Download the “Low rew multi-view, training data, all distorted images” here and unzip them in multiview_training/
• Download the “Low rew multi-view, testing data, all undistorted images” here and unzip them in multiview_testing/
• We directly provide correspondences for pairs of images taken at different intervals. There is one bundle file for each dataset and each rate of interval, for example “lakeside_every_5_rate_of_3”. This means that we sampled the source images every 5 images and the target image is taken at a particular rate from each source image. Download all these files here and unzip them.

As illustration, your root ETH3D directory should be organised as follows:

/ETH3D/
       multiview_testing/
                        lakeside/
                        sand_box/
                        storage_room/
                        storage_room_2/
                        tunnel/
       multiview_training/
                        delivery_area/
                        electro/
                        forest/
                        playground/
                        terrains/
        info_ETH3D_files/

The organisation of your directories is important, since the bundle files contain the relative paths to the images, from the ETH3D root folder.

Evaluation: for each interval rate (3,5,7,9,11,13,15), we compute the metrics for each of the sub-datasets (lakeside, delivery area and so on). The final metrics are the average over all datasets for each rate. After updating the path of ‘eth3d’ in admin/local.py, evaluation is run with

python eval_matching.py --datasets robotcar --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir PDCNet --multi_stage_type direct

AEPE for different rates of intervals between image pairs.

PCK-1 for different rates of intervals between image pairs:

Note that the PCKs are computed per image, and then averaged per sequence. The final metrics is the average over all sequences. It corresponds to the results ‘_per_image’ in the outputted metric file. Note that this is not the metrics used in the PDC-Net paper, where the PCKs are c omputed per sequence instead, using the PDC-Net direct approach (corresponds to results ‘_per_dataset’ in outputted metric file).

PCK-5 for different rates of intervals between image pairs:

Data preparation: Download the data with

bash assets/download_hpatches.sh

The corresponding csv files for each viewpoint ID with the path to the images and the homography parameters relating the pairs are listed in assets/.

Evaluation: After updating the path of ‘hp’ in admin/local.py, evaluation is run with

python eval_matching.py --datasets hp --model GLUNet_GOCor --pre_trained_models static --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir

Similar results should be obtained:

Data preparation: Both KITTI-2012 and 2015 datasets are available here

Evaluation: After updating the path of ‘kitti2012’ and ‘kitti2015’ in admin/local.py, evaluation is run with

python eval_matching.py --datasets kitti2015 --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 PDCNet --multi_stage_type direct

Similar results should be obtained:

Data preparation: Download the data with

bash assets/download_sintel.sh

Evaluation: After updating the path of ‘sintel’ in admin/local.py, evaluation is run with

python eval_matching.py --datasets sintel --model PDCNet --pre_trained_models megadepth --optim_iter 3 --local_optim_iter 7 --save_dir path_to_save_dir PDCNet --multi_stage_type direct

Similar results should be obtained:

Data preparation: To download the images, run:

bash assets/download_tss.sh

Evaluation: After updating the path of ‘tss’ in admin/local.py, evaluation is run with

python eval_matching.py --datasets TSS --model GLUNet_GOCor --pre_trained_models static --optim_iter 3 --local_optim_iter 7 --flipping_condition True --save_dir path_to_save_dir

Similar results should be obtained:

Data preparation: To download the images, run:

bash assets/download_pf_pascal.sh

Evaluation: After updating the path of ‘PFPascal’ in admin/local.py, evaluation is run with

python eval_matching.py --datasets PFPascal --model GLUNet_GOCor --pre_trained_models static --optim_iter 3 --local_optim_iter 7 --flipping_condition True --save_dir path_to_save_dir

Data preparation: To download the images, run:

bash assets/download_pf_willow.sh

Evaluation: After updating the path of ‘PFWillow’ in admin/local.py, evaluation is run with

python eval_matching.py --datasets PFWillow --model GLUNet_GOCor --pre_trained_models static --optim_iter 3 --local_optim_iter 7 --flipping_condition True --save_dir path_to_save_dir

Data preparation: To download the images, run:

bash assets/download_spair.sh

Evaluation: After updating the path of ‘spair’ in admin/local.py, evaluation is run with

python eval_matching.py --datasets spair --model GLUNet_GOCor --pre_trained_models static --optim_iter 3 --local_optim_iter 7 --flipping_condition True --save_dir path_to_save_dir

Data preparation: The groundtruth for YFCC is provided the file assets/yfcc_test_pairs_with_gt_original.txt (from SuperGlue repo). Images can be downloaded from the OANet repo and moved to the desired location

bash assets/download_yfcc.sh

File structure should be

YFCC
└──  images/
       ├── buckingham_palace/
       ├── notre_dame_front_facade/
       ├── reichstag/
       └── sacre_coeur/

Evaluation: After updating the path ‘yfcc’ in admin/local.py, compute metrics on YFCC100M with PDC-Net multiscale (MS) using the command:

python -u eval_pose_estimation.py --datasets YFCC --model PDCNet --pre_trained_models megadepth --optim_iter 3  --local_optim_iter 7 --estimate_at_quarter_reso True --mask_type_for_pose_estimation proba_interval_1_above_10 --save_dir path_to_save_dir --plot False PDCNet --multi_stage_type multiscale_homo_from_quarter_resolution_uncertainty --mask_type proba_interval_1_above_10 

You should get similar metrics (not exactly the same because of RANSAC):

Data preparation: Go to the ScanNet github repo to download the ScanNet test set (100 scenes). You will need to extract the raw sensor data from the 100 .sens files in each scene in the test set using the SensReader tool. We use the groundtruth provided by in the SuperGlue repo provided here in the file assets/scannet_test_pairs_with_gt.txt.

Evaluation: After updating the path ‘scannet_test’ in admin/local.py, compute metrics on ScanNet with PDC-Net multiscale (MS) using the command:

python -u eval_pose_estimation.py --datasets scannet --model PDCNet --pre_trained_models megadepth --optim_iter 3  --local_optim_iter 7 --estimate_at_quarter_reso True --mask_type_for_pose_estimation proba_interval_1_above_10 --save_dir path_to_save_dir --plot False PDCNet --multi_stage_type multiscale_homo_from_quarter_resolution_uncertainty --mask_type proba_interval_1_above_10 

You should get similar metrics (not exactly the same because of RANSAC):

This is the same image pairs used in GLU-Net repo. For the training, we use a combination of the DPED, CityScapes and ADE-20K datasets. The DPED training dataset is composed of only approximately 5000 sets of images taken by four different cameras. We use the images from two cameras, resulting in around 10,000 images. CityScapes additionally adds about 23,000 images. We complement with a random sample of ADE-20K images with a minimum resolution of 750 x 750. It results in 40.000 original images, used to create pairs of training images by applying geometric transformations to them. The path to the original images as well as the geometric transformation parameters are given in the csv files ‘assets/csv_files/homo_aff_tps_train_DPED_CityScape_ADE.csv’ and ‘assets/csv_files/homo_aff_tps_test_DPED_CityScape_ADE.csv’.

Apparently, the structure of the ADE-20K dataset has changed and the provided paths in the csv files are not valid anymore. I am working on a fix for the ADE-20K images. In the meantime, use ‘assets/csv_files/homo_aff_tps_train_DPED_CityScape.csv’ and ‘assets/csv_files/homo_aff_tps_test_DPED_CityScape.csv’ to exclude the ADE images (you don’t need to download the ADE-20K dataset in that case). Resulting training data is 31K images. Performance of the resulting trained model might be a bit different.

1. Download the original images

• Download the DPED dataset (54 GB) ==> images are created in original_images/

• Download the CityScapes dataset

  • download ‘leftImg8bit_trainvaltest.zip’ (11GB, left 8-bit images – train, val, and test sets’, 5000 images) ==> images are created in CityScape/
  • download leftImg8bit_trainextra.zip (44GB, left 8-bit images – trainextra set, 19998 images) ==> images are created in CityScape_extra/

• Download the ADE-20K dataset (3.8 GB, 20.210 images) ==> images are created in ADE20K_2016_07_26/

Put all the datasets in the same directory. As illustration, your root training directory should be organised as follows:

training_datasets/
    ├── original_images/
    ├── CityScape/
    ├── CityScape_extra/
    └── ADE20K_2016_07_26/

2. Save the synthetic image pairs and flows to disk
   During training, from this set of original images, the pairs of synthetic images could be created on the fly at each epoch. However, this dataset generation takes time and since no augmentation is applied at each epoch, one can also create the dataset in advance and save it to disk. During training, the image pairs composing the training datasets are then just loaded from the disk before passing through the network, which is a lot faster. To generate the training dataset and save it to disk:

python assets/save_training_dataset_to_disk.py --image_data_path /directory/to/original/training_datasets/ 
--csv_path assets/homo_aff_tps_train_DPED_CityScape_ADE.csv --save_dir /path/to/save_dir --plot True

It will create the images pairs and corresponding flow fields in save_dir/images and save_dir/flow respectively.

3. Add the paths in admin/local.py as ‘training_cad_520’ and ‘validation_cad_520’

This is useful for adding moving objects. Download the images along with annotations from here. The root folder should be organized as follows. The add the paths in admin/local.py as ‘coco’.

coco_root
    └── annotations
        ├── instances_train2014.json
        └── instances_train2017.json
    └──images
        ├── train2014
        └── train2017

We use the reconstructions provided in the D2-Net repo. You can download the undistorted reconstructions and aggregated scene information folder directly here – Google Drive.

File structure should be the following:

MegaDepth
├── Undistorted_Sfm
└── scene_info

Them add the paths in admin/local.py as ‘megadepth_training’.

• PDCNet.PDCNet_stage1: The default settings used for first stage network training with fixed backbone weights. We initialize the backbone VGG-16 with pre-trained ImageNet weights. We train first on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), on which we add independently moving objects and perturbations.

• PDCNet.PDCNet_stage2: The default settings used for training the final PDC-Net model. This setting fine-tunes all layers in the model trained using PDCNet_stage1 (including the feature backbone). As training dataset, we use a combination of the same dataset than in stage 1 as well as image pairs from the MegaDepth dataset and their sparse ground-truth correspondence data.

• PDCNet.GLUNet_GOCor_star_stage1: Same settings than for PDCNet_stage1, with different model (non probabilistic baseline). The loss is changed accordingly to the L1 loss instead of the negative log likelihood loss.

• PDCNet.GLUNet_GOCor_star_stage2: The default settings used for training the final GLU-Net-GOCor* (see PDCNet paper).

• GLUNet.GLUNet_static: The default settings used training the final GLU-Net (of the paper GLU-Net).
  We fix the backbone weights and initialize the backbone VGG-16 with pre-trained ImageNet weights. We train on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), which is later (GOCor paper) referred to as ‘static’ dataset.

• GLUNet.GLUNet_dynamic: The default settings used training the final GLU-Net trained on the dynamic dataset (of the paper GOCor).
  We fix the backbone weights and initialize the backbone VGG-16 with pre-trained ImageNet weights. We train on synthetically generated image pairs from the DPED, CityScape and ADE dataset (pre-computed and saved), on which we add one independently moving object. This dataset is referred to as ‘dynamic’ dataset in GOCor paper.