Zero and R2D2: A Large-scale Chinese Cross-modal Benchmark and A Vision-Language Framework

This repo is the official implementation of R2D2. It includes datasets, code, and models as following:

✅Zero benchmark is available. The detailed introduction and download URL are in http://zero.so.com.

✅Pre-trained checkpoints.

✅Inference demo.

❌Fine-tuning code and checkpoints for Image-Text Retrieval and Image-Text Matching tasks (coming soon).

❌Pre-training code (coming soon).

Requirements

pip install -r requirements.txt

Pre-trained checkpoints

Pre-trained image-text pairs R2D2ViT-L PRD2ViT-L
250M Download Download
23M Download –
2.3M Download –

Inference demo

  • To evaluate the pretrained R2D2 model on image-text pairs, run: 
    python r2d2_inference_demo.py
  • To evaluate the pretrained PRD2 model on image-text pairs, run: 
    python prd2_inference_demo.py

Citation

If you find this code to be useful for your research, please consider citing.

@article{xie2022zero,
  title={Zero and R2D2: A Large-scale Chinese Cross-modal Benchmark and A Vision-Language Framework},
  author={Xie, Chunyu and Cai, Heng and Song, Jianfei and Li, Jincheng and Kong, Fanjing and Wu, Xiaoyu and Morimitsu, Henrique and Yao, Lin and Wang, Dexin and Zhang, Xiangzheng and Leng, Dawei and Ji, Xiangyang and Deng, Yafeng },
  journal={arXiv preprint arXiv:2205.03860},
  year={2022}
}

GitHub

View Github