Video Demoireing with Relation-Based Temporal Consistency

Video Demoireing with Relation-Based Temporal Consistency (CVPR 2022) Peng Dai, Xin Yu, Lan Ma, Baoheng Zhang, Jia Li, Wenbo Li, Jiajun Shen, Xiaojuan Qi. Paper, Project_page, Video

Introduction

Our video demoireing network mainly consists of three parts: First, the PCD takes consecutive frames as inputs to implicitly align frames in the feature space. Second, the feature aggregation module merges aligned frame features at different scales by predicting blending weights. Third, the merged features are sent to the demoire model with dense connections to realize moire artifacts removal.

Dataset

We constrcut the first video demoireing dataset, containing 290 videos (720p, 60 frames for each video), and the equipments we adpoted are ‘TCL20 pro mobile phone + Huipu v270 monitor’ and ‘iPhoneXR + MackBook Pro’.

In data_v1, image pairs are aligned using the homography. In data_v2, aligned image pairs are refined using the estimated optical flow. Besides, we also provide the original moire videos.

Please download and unzip the dataset you need, then change the data paths in ‘config/video_demoire_temporal_mbr_scratch_v2.yaml’ line 7 and 8. Note that the results in our paper are based on the data_v1.

Environments

Create the conda virtual environment (python 3.7) and activate it. conda create -n vdm python=3.7 conda activate vdm

Intsall pytorch 1.6.0 and other libaries in the ‘requirements.txt’. conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.2 -c pytorch pip install -r requirements.txt

Install the DCNv2. cd model/DCNv2 python3 setup.py build develop

Our implementation has been tested on one NVIDIA 2080Ti GPU with cuda 10.2.

Train

To train the model, just run: python train.py --config config/video_demoire_temporal_mbr_scratch_v2.yaml

Note: if too large offsets appear while training the PCD model, please restart training. Alternatively, you can use this model (train two epoches) to initialize the network.

Test

To test the model, just run: python test.py --config config/video_demoire_temporal_mbr_scratch_v2.yaml

We also provide pre-trained models for both TCL and iPhone. The ‘checkpoint_000049.tar’ means no realtion-based loss, and ‘checkpoint_000059.tar’ means with realtion-based loss. Please download these models into the corresponding folder (line 29) and change the test epoch (line 54) as descirbed in ‘config/video_demoire_temporal_mbr_scratch_v2.yaml’.

Results

Contact

If you have any questions, you can email me ([email protected]).

Citation

If you are interested in our code or method, please consider citing the following:

@inproceedings{dai2022video,
  title={Video Demoireing with Relation-Based Temporal Consistency},
  author={Dai, Peng and Yu, Xin and Ma, Lan and Zhang, Baoheng and Li, Jia and Li, Wenbo and Shen, Jiajun and Qi, Xiaojuan},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2022}
}

GitHub

View Github