PPDM
Code for our CVPR 2020 paper "PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection".
Contributed by Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, Jiashi Feng.
Checklist
- [x] Training code and test code on HICO-Det dataset. (2020-03-11)
- [x] Training code and test code on HOI-A dataset. (2020-03-11)
- [ ] HOI-A dataset.
- [ ] Image demo.
- [ ] Video demo.
- [ ] PPDM for video HOI detection.
- [ ] PPDM for human-centric relationship segmentation.
Getting Started
Installation
-
Clone this repository.
git clone https://github.com/YueLiao/PPDM.git $PPDM_ROOT
-
Install pytorch0.4.1.
conda install pytorch=0.4.1 torchvision -c pytorch
-
Install the requirements.
pip install -r requirements.txt
-
Compile deformable convolutional (from DCNv2).
cd $PPDM_ROOT/src/lib/models/networks/DCNv2 ./make.sh
Demo
-
Image Demo
-
Video Demo
Training and Test
Dataset Preparation
-
Download HICO-Det datasets. Organize them in
Dataset
folder as follows:|-- Dataset/ | |-- <dataset name>/ | |-- images | |-- annotations
-
Download the pre-processed annotations for HICO-Det from the [websit] and replace the original annotations in
Dataset
folder. The pre-processed annotations including|-- anotations/ | |-- trainval_hico.json | |-- test_hico.json | |-- corre_hico.npy
The
trainval_hico.json
andtest_hico.json
are the "HOI-A format" annotations generated from iCAN annotation.corre_hico.npy
is a binary mask, if theith category of object
and thejth category of verb
can form an HOI label, the value at location (i, j) ofcorre_hico.npy
is set to 1, else 0.
Training
-
Download the corresponding pre-trained models trained on COCO object detection dataset provided by CenterNet. (Res18, DLA34, Hourglass104). Put them into the
models
folder. -
The scripts for training in
experiments
folder. An example traning on HICO-DET dataset as follow:cd src python main.py --batch_size 112 --master_batch 7 --lr 4.5e-4 --gpus 0,1,2,3,4,5,6,7 --num_workers 16 --load_model ../models/ctdet_coco_dla_2x.pth --image_dir images/train2015 --dataset hico --exp_id hoidet_hico_dla
Test
-
Evalution by our rewritten script and select the best checkpoint. The scripts for evalution are put into
experiments
folder. An example evalution on HICO-DET dataset as follow:cd src python test_hoi.py --exp_id hoidet_hico_dla --gpus 0 --dataset hico --image_dir images/test2015 --test_with_eval
or directly generating the predictions and evalutating based on our provided checkpoint:
cd src python test_hoi.py --exp_id hoidet_hico_dla --load_model ../exp/hoidet/hoidet_hico_dla/hoidet_hico_dla.pth --gpus 0 --dataset hico --image_dir images/test2015 --test_with_eval
-
For HICO-DET official evalution.
-
Setup HICO-DET evaluation code:
cd src/lib/eval sh set_hico_evalution.sh
-
Evaluate your prediction:
cd src/lib/eval python trans_for_eval_hico.py best_predictions.json cd ho-rcnn matlab -r "Generate_detection.m; quite"
Results on HICO-DET and HOI-A
Our Results on HICO-DET dataset
Model | Full (def) | Rare (def) | None-Rare (def) | Full (ko) | Rare (ko) | None-Rare (ko) | FPS | Download |
---|---|---|---|---|---|---|---|---|
res18 | 14.94 | 8.83 | 16.76 | 16.98 | 10.42 | 18.94 | 89 | model |
dla34 | 19.34 | 12.21 | 21.47 | 21.53 | 14.14 | 23.73 | 38 | model |
dla34_3level | 19.33 | 11.7 | 21.61 | 21.73 | 14.05 | 24.03 | 37 | model |
dla34_glob | 19.42 | 13.12 | 21.3 | 21.76 | 15.13 | 23.74 | 38 | model |
dla34_glob_3level | 19.75 | 12.38 | 21.95 | 22.13 | 14.72 | 24.35 | 37 | model |
hourglass104 | 21.92 | 15.13 | 23.95 | 24.25 | 17.21 | 26.35 | 14 | model |
Our Results on HOI-A dataset
Coming soon.
Citation
Please consider citing this project in your publications if it helps your research. The following is a BibTeX reference. The BibTeX entry requires the url LaTeX package.
@article{liao2019ppdm,
title={PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection},
author={Liao, Yue and Liu, Si and Wang, Fei and Chen, Yanjie and Qian, Chen and Feng, Jiashi},
journal={arXiv preprint arXiv:1912.12898},
year={2019}
}