A simple and effective query based instance segmentation method driven

Instances as Queries

120617230-7d34a600-c48c-11eb-8a43-d61689a050be

TL;DR: QueryInst (Instances as Queries) is a simple and effective query based instance segmentation method driven by parallel supervision on dynamic mask heads, which outperforms previous arts in terms of both accuracy and speed.
Our QueryTrack (i.e., Tracking Instances as Queries, tech report) based on QueryInst won the 2nd place (AP = 52.3 @ test set, AP = 54.3 @ val set) in video instance segmentation (VIS) track with single online end-to-end model, single scale testing & without using extra video training data in the 3rd Large-scale Video Object Segmentation Challenge, CVPR 2021.
For the first time, we demonstrate that an end-to-end query based framework driven by parallel supervision is competitive with well-established and highly-optimized methods in a wide range of instance-level recognition tasks (object detection, instance segmentation and video instance segmentation).

Instances as Queries

by Yuxin Fang*, Shusheng Yang*, Xinggang Wang†, Yu Li, Chen Fang, Ying Shan, Bin Feng, Wenyu Liu.

(*) equal contribution, (†) corresponding author.

arXiv technical report (arXiv 2105.01928)

QueryInst

This repo serves as the official implementation for QueryInst, based on mmdetection and built upon Sparse R-CNN & DETR. Implantations based on Detectron2 will be released in the near future.
This project is under active development, we will extend QueryInst to a wide range of instance-level recognition tasks.

Main Results on COCO test-dev

Configs	Aug.	Weights	Box AP	Mask AP
QueryInst_Swin_L_300_queries (single scale testing)	400 ~ 1200, w/ Crop	baidu / google	56.1	49.1

Main Results on COCO val

Configs	Aug.	Weights	Box AP	Mask AP
QueryInst_R50_3x_300_queries	480 ~ 800, w/ Crop	baidu / google	46.9	41.4
QueryInst_R101_3x_300_queries	480 ~ 800, w/ Crop	baidu / google	48.0	42.4
QueryInst_X101-DCN_3x_300_queries	480 ~ 800, w/ Crop	-	50.3	44.2
QueryInst_Swin_L_300_queries (single scale testing)	400 ~ 1200, w/ Crop	baidu / google	56.1	48.9

Notes:

Accesscode for baidu is QIst.

Getting Started

Our project is mainly developed on mmdetection toolbox (931d96), please refer to the mmdetection official installation.
Install QueryInst by:

python setup.py develop

Prepare datasets:

mkdir data && cd data
ln -s /path/to/coco coco

Training QueryInst with single GPU:

python tools/train.py configs/queryinst/queryinst_r50_fpn_1x_coco.py

Training QueryInst with multi GPUs:

./tools/dist_train.sh configs/queryinst/queryinst_r50_fpn_1x_coco.py 8

Test QueryInst on COCO val set with single GPU:

python tools/test.py configs/queryinst/queryinst_r50_fpn_1x_coco.py PATH/TO/CKPT.pth --eval bbox segm

Test QueryInst on COCO val set with multi GPUs:

./tools/dist_test.sh configs/queryinst/queryinst_r50_fpn_1x_coco.py PATH/TO/CKPT.pth 8 --eval bbox segm

Citation

If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil: :

@article{QueryInst,
  title={Instances as Queries},
  author={Fang, Yuxin and Yang, Shusheng and Wang, Xinggang and Li, Yu and Fang, Chen and Shan, Ying and Feng, Bin and Liu, Wenyu},
  journal={arXiv preprint arXiv:2105.01928},
  year={2021}
}

@article{QueryTrack,
  title={Tracking Instances as Queries},
  author={Yang, Shusheng and Fang, Yuxin and Wang, Xinggang and Li, Yu and Shan, Ying and Feng, Bin and Liu, Wenyu},
  journal={arXiv preprint arXiv:2106.11963},
  year={2021}
}