Pytorch implementation for the Dynamic Concept Learner

DCL-PyTorch

Pytorch implementation for the Dynamic Concept Learner (DCL).

Prerequisites

• Python 3
• PyTorch 1.0 or higher, with NVIDIA CUDA Support
• Other required python packages specified by requirements.txt. See the Installation.

Installation

Install Jacinle: Clone the package, and add the bin path to your global PATH environment variable:

git clone https://github.com/vacancy/Jacinle --recursive
export PATH=<path_to_jacinle>/bin:$PATH

Clone this repository:

git clone https://github.com/zfchenUnique/DCL-Release.git --recursive

Create a conda environment for NS-CL, and install the requirements. This includes the required python packages
from both Jacinle NS-CL. Most of the required packages have been included in the built-in anaconda package:

Dataset preparation

• Download videos, video annotation, questions and answers, and object proposals accordingly from the official website
• Transform videos into ".png" frames with ffmpeg.
• Organize the data as shown below.

  clevrer
  ├── annotation_00000-01000
  │   ├── annotation_00000.json
  │   ├── annotation_00001.json
  │   └── ...
  ├── ...
  ├── image_00000-01000
  │   │   ├── 1.png
  │   │   ├── 2.png
  │   │   └── ...
  │   └── ...
  ├── ...
  ├── questions
  │   ├── train.json
  │   ├── validation.json
  │   └── test.json
  ├── proposals
  │   ├── proposal_00000.json
  │   ├── proposal_00001.json
  │   └── ...
  

Fast Evaluation

• Download the extracted object trajectories from google drive.
• Git clone the dynamic model, download image proposals and the pretrained propNet models and make dynamic prediction by

    git clone https://github.com/zfchenUnique/clevrer_dynamic_propnet.git
    cd clevrer_dynamic_propnet
    sh ./scripts/eval_fast_release_v2.sh 0

• Download the pretrained DCL model and parsed programs.

   sh scripts/script_test_prp_clevrer_qa.sh 0

• Get the accuracy on evalAI.

Step-by-step Training

• Step 1: download the proposals from the region proposal network and extract object trajectories for train and val set by

   sh scripts/script_gen_tubes.sh

• Step 2: train a concept learner with descriptive and explanatory questions for static concepts (i.e. color, shape and material)

   sh scripts/script_train_dcl_stage1.sh 0

• Step 3: extract static attributes & refine object trajectories
  extract static attributes

   sh scripts/script_extract_attribute.sh

refine object trajectories

   sh scripts/script_gen_tubes_refine.sh

• Step 4: extract predictive and counterfactual scenes by

    cd clevrer_dynamic_propnet
    sh ./scripts/train_tube_box_only.sh # train
    sh ./scripts/train_tube.sh # train
    sh ./scripts/eval_fast_release_v2.sh 0 # val

• Step 5: train DCL with all questions and the refined trajectories

   sh scripts/script_train_dcl_stage2.sh 0

Generalization to CLEVRER-Grounding

• Step 1: download expression annotation and parsed programs from google drive
• Step 2: evaluate the performance on CLEVRER-Grounding

    sh ./scripts/script_grounding.sh  0
    jac-crun 0 scripts/script_evaluate_grounding.py

Generalization to CLEVRER-Retrieval

• Step 1: download expression annotation and parsed programs from google drive
• Step 2: evaluate the performance on CLEVRER-Retrieval

    sh ./scripts/script_retrieval.sh  0
    jac-crun 0 scripts/script_evaluate_retrieval.py

Extension to Tower Blocks

• Step 1: download question annotation and videos from google drive
• Step 2: train on Tower block QA

    sh ./scripts/script_train_blocks.sh 0

• Step 3: download the pretrain model from google drive and evaluate on Tower block QA

    sh ./scripts/script_eval_blocks.sh 0

GitHub

https://github.com/zfchenUnique/DCL-Release