FID calculation with proper image resizing and quantization steps

clean-fid for Evaluating Generative Models

The FID calculation involves many steps that can produce inconsistencies in the final metric. As shown below, different implementations use different low-level image quantization and resizing functions, the latter of which are often implemented incorrectly.

We provide an easy-to-use library to address the above issues and make the FID scores comparable across different methods, papers, and groups.

On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation

Gaurav Parmar, Richard Zhang, Jun-Yan Zhu

arXiv 2104.11222, 2021

CMU and Adobe

CleanFID Leaderboard API

We compute the FID scores using the corresponding methods used in the original papers and using the Clean-FID proposed here.
All values are computed using 10 evaluation runs.

We provide an API to query the results shown in the tables below directly from
the pip package. The arguments model_name, dataset_name, dataset_res, dataset_split, task_name can be used to filter
the results.

Buggy Resizing Operations

The definitions of resizing functions are mathematical and should never be a function of the library being used. Unfortunately, implementations differ across commonly-used libraries. They are often implemented incorrectly by popular libraries. Try out the different resizing implementations in the Google colab notebook here.

The inconsistencies among implementations can have a drastic effect of the evaluations metrics. The table below shows that FFHQ dataset images resized with bicubic implementation from other libraries (OpenCV, PyTorch, TensorFlow, OpenCV) have a large FID score (≥ 6) when compared to the same images resized with the correctly implemented PIL-bicubic filter. Other correctly implemented filters from PIL (Lanczos, bilinear, box) all result in relatively smaller FID score (≤ 0.75). Note that since TF 2.0, the new flag antialias (default: False) can produce results close to PIL. However, it was not used in the existing TF-FID repo and set as False by default.

JPEG Image Compression

Image compression can have a surprisingly large effect on FID. Images are perceptually indistinguishable from each other but have a large FID score. The FID scores under the images are calculated between all FFHQ images saved using the corresponding JPEG format and the PNG format.

Below, we study the effect of JPEG compression for StyleGAN2 models trained on the FFHQ dataset (left) and LSUN outdoor Church dataset (right). Note that LSUN dataset images were collected with JPEG compression (quality 75), whereas FFHQ images were collected as PNG. Interestingly, for LSUN dataset, the best FID score (3.48) is obtained when the generated images are compressed with JPEG quality 87.

Quick Start

• install requirements

  pip install -r requirements.txt
  

• install the library

  pip install clean-fid
  

Computing FID

• Compute FID between two image folders

  from cleanfid import fid
  score = fid.compute_fid(fdir1, fdir2)
  

• Compute FID between one folder of images and pre-computed datasets statistics (e.g., FFHQ)

  from cleanfid import fid
  score = fid.compute_fid(fdir1, dataset_name="FFHQ", dataset_res=1024, dataset_split="trainval70k")
  

• Compute FID using a generative model and pre-computed dataset statistics:

  from cleanfid import fid
  # function that accepts a latent and returns an image in range[0,255]
  gen = lambda z: GAN(latent=z, ... , <other_flags>)
  score = fid.compute_fid(gen=gen, dataset_name="FFHQ",
          dataset_res=256, num_gen=50_000, dataset_split="trainval70k")
  

Computing KID

The KID score can be computed using a similar interface as FID.
The dataset statistics for KID are only precomputed for smaller datasets AFHQ, BreCaHAD, and MetFaces.

• Compute KID between two image folders

  from cleanfid import fid
  score = fid.compute_kid(fdir1, fdir2)
  

• Compute KID between one folder of images and pre-computed datasets statistics

  from cleanfid import fid
  score = fid.compute_kid(fdir1, dataset_name="brecahad", dataset_res=512, dataset_split="train")
  

• Compute KID using a generative model and pre-computed dataset statistics:

  from cleanfid import fid
  # function that accepts a latent and returns an image in range[0,255]
  gen = lambda z: GAN(latent=z, ... , <other_flags>)
  score = fid.compute_kid(gen=gen, dataset_name="brecahad", dataset_res=512, num_gen=50_000, dataset_split="train")
  

Supported Precomputed Datasets

We provide precompute statistics for the following commonly used configurations. Please contact us if you want to add statistics for your new datasets.

Using precomputed statistics
In order to compute the FID score with the precomputed dataset statistics, use the corresponding options. For instance, to compute the clean-fid score on generated 256x256 FFHQ images use the command:

fid_score = fid.compute_fid(fdir1, dataset_name="ffhq", dataset_res=256,  mode="clean", dataset_split="trainval70k")

Create Custom Dataset Statistics

• dataset_path: folder where the dataset images are stored

• custom_name: name to be used for the statistics

• Generating custom statistics (saved to local cache)

  from cleanfid import fid
  fid.make_custom_stats(custom_name, dataset_path, mode="clean")
  

• Using the generated custom statistics

  from cleanfid import fid
  score = fid.compute_fid("folder_fake", dataset_name=custom_name,
            mode="clean", dataset_split="custom")
  

• Removing the custom stats

  from cleanfid import fid
  fid.remove_custom_stats(custom_name, mode="clean")
  

• Check if a custom statistic already exists

  from cleanfid import fid
  fid.test_stats_exists(custom_name, mode)
  

Backwards Compatibility

We provide two flags to reproduce the legacy FID score.

• mode="legacy_pytorch"
  
  This flag is equivalent to using the popular PyTorch FID implementation provided here
  
  
  The difference between using clean-fid with this option and code is ~2e-06
  
  
  See doc for how the methods are compared

• mode="legacy_tensorflow"
  
  This flag is equivalent to using the official implementation of FID released by the authors.
  
  
  The difference between using clean-fid with this option and code is ~2e-05
  
  
  See doc for detailed steps for how the methods are compared

Building clean-fid locally from source

python setup.py bdist_wheel
pip install dist/*

CleanFID Leaderboard for common tasks

We compute the FID scores using the corresponding methods used in the original papers and using the Clean-FID proposed here.
All values are computed using 10 evaluation runs. We provide an API to query the results shown in the tables below directly from the pip package.

If you would like to add new numbers and models to our leaderboard, feel free to contact us.

CIFAR-10 (few shot)

The test set is used as the reference distribution and compared to 10k generated images.

100% data

20% data

10% data

CIFAR-100 (few shot)

The test set is used as the reference distribution and compared to 10k generated images.

100% data

20% data

10% data

FFHQ

all images @ 1024x1024

Values are computed using 50k generated images

30k - images @ 256x256 (Few Shot Generation)

The 70k images from trainval70k set is used as the reference images and compared to 50k generated images.

10k - images @ 256x256 (Few Shot Generation)

The 70k images from trainval70k set is used as the reference images and compared to 50k generated images.

5k - images @ 256x256 (Few Shot Generation)

The 70k images from trainval70k set is used as the reference images and compared to 50k generated images.

1k - images @ 256x256 (Few Shot Generation)

The 70k images from trainval70k set is used as the reference images and compared to 50k generated images.

LSUN Categories

100% data

The 50k images from train set is used as the reference images and compared to 50k generated images.

LSUN CAT - 30k images (Few Shot Generation)

All 1,657,264 images from trainfull split are used as the reference images and compared to 50k generated images.

LSUN CAT - 10k images (Few Shot Generation)

All 1,657,264 images from trainfull split are used as the reference images and compared to 50k generated images.

LSUN CAT - 5k images (Few Shot Generation)

All 1,657,264 images from trainfull split are used as the reference images and compared to 50k generated images.

LSUN CAT - 1k images (Few Shot Generation)

All 1,657,264 images from trainfull split are used as the reference images and compared to 50k generated images.

AFHQ (Few Shot Generation)

AFHQ Dog

All 4739 images from train split are used as the reference images and compared to 50k generated images.

AFHQ Wild

All 4738 images from train split are used as the reference images and compared to 50k generated images.

BreCaHAD (Few Shot Generation)

All 1944 images from train split are used as the reference images and compared to 50k generated images.

MetFaces (Few Shot Generation)

All 1336 images from train split are used as the reference images and compared to 50k generated images.

Horse2Zebra (Image to Image Translation)

All 140 images from test split are used as the reference images and compared to 120 translated images.

† The translated images for these methods were intitially compared by [Park et al, 2020] using .jpeg compressions.
We retrained the using these models using the same settings and generated the images as .png for a fair comparision

Cat2Dog (Image to Image Translation)

All 500 images from test split are used as the reference images and compared to 500 translated images.

Citation

If you find this repository useful for your research, please cite the following work.

@article{parmar2021cleanfid,
  title={On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation},
  author={Parmar, Gaurav and Zhang, Richard and Zhu, Jun-Yan},
  journal={arXiv preprint arXiv:2104.11222},
  year={2021}
}

GitHub

https://github.com/GaParmar/clean-fid