Facial Expression Recognition using Residual Masking Network, in PyTorch.

A PyTorch implementation of my thesis with the same name.

Live Demo:

Approach 1:

  1. Install from pip
pip install rmn
  1. Run video demo by the following Python scripts
from rmn import video_demo


Approach 2:

  1. clone the repo and install package via pip
git clone [email protected]:phamquiluan/ResidualMaskingNetwork.git 

cd ResidualMaskingNetwork
pip install -e .
  1. call video_demo in rmn package
from rmn import video_demo

Approach 3:

  • Model file: download (this checkpoint is trained on VEMO dataset, locate it at ./saved/checkpoints/ directory)
  • Download 2 files: prototxt, and res10_300x300_ssd for face detection OpenCV. Locate at current directory or checking file path with file.


Benchmarking on FER2013

We benchmark our code thoroughly on two datasets: FER2013 and VEMO. Below are the results and trained weights:

Model Accuracy
VGG19 70.80
EfficientNet_b2b 70.80
Googlenet 71.97
Resnet34 72.42
Inception_v3 72.72
Bam_Resnet50 73.14
Densenet121 73.16
Resnet152 73.22
Cbam_Resnet50 73.39
ResMaskingNet 74.14
ResMaskingNet + 6 76.82

Results in VEMO dataset could be found in my thesis or slide (attached below)

Benchmarking on ImageNet

We also benchmark our model on ImageNet dataset.

Model Top-1 Accuracy Top-5 Accuracy
Resnet34 72.59 90.92
CBAM Resnet34 73.77 91.72
ResidualMaskingNetwork 74.16 91.91


  • Install PyTorch by selecting your environment on the website and running the appropriate command.
  • Clone this repository and install package prerequisites below.
  • Then download the dataset by following the instructions below.


  • Python 3.6+
  • PyTorch 1.3+
  • Torchvision 0.4.0+
  • requirements.txt


Training on FER2013

  • To train network, you need to specify model name and other hyperparameters in config file (located at configs/*) then ensure it is loaded in main file, then run training procedure by simply run main file, for example:
python  # Example for fer2013_config.json file
  • The best checkpoints will chosen at term of best validation accuracy, located at saved/checkpoints
  • The TensorBoard training logs are located at saved/logs, to open it, use tensorboard --logdir saved/logs/


  • By default, it will train alexnet model, you can switch to another model by edit configs/fer2013\_config.json file (to resnet18 or cbam\_resnet50 or my network resmasking\_dropout1.

Training on Imagenet dataset

To perform training resnet34 on 4 V100 GPUs on a single machine:

python ./ -a resnet34 --dist-url 'tcp://' --dist-backend 'nccl' --multiprocessing-distributed --world-size 1 --rank 0 


For student, who takes care of font family of confusion matrix and would like to write things in LaTeX, below is an example for generating a striking confusion matrix.

(Read this article for more information, there will be some bugs if you blindly run the code without reading).



Ensemble method

I used no-weighted sum avarage ensemble method to fusing 7 different models together, to reproduce results, you need to do some steps:

  1. Download all needed trained weights and located on ./saved/checkpoints/ directory. Link to download can be found on Benchmarking section.
  2. Edit file gen_results and run it to generate result offline for each model.
  3. Run file to generate accuracy for example methods.

Dissertation and Slide


We have accumulated the following to-do list, which we hope to complete in the near future

  • Still to come:
    • [x] Upload all models and training code.
    • [x] Test time augmentation.
    • [x] GPU-Parallel.
    • [x] Pretrained model.
    • [x] Demo and inference code.
    • [x] Imagenet trained and pretrained weights.
    • [ ] GradCAM visualization and Pooling method for visualize activations.
    • [ ] Centerloss Visualizations.


Note: Unfortunately, I am currently join a full-time job and research on another topic, so I'll do my best to keep things up to date, but no guarantees. That being said, thanks to everyone for your continued help and feedback as it is really appreciated. I will try to address everything as soon as possible.


  • Same as in dissertation.


Author = {Luan Pham & Tuan Anh Tran},
Title = {Facial Expression Recognition using Residual Masking Network},
url = {},
Year = {2020}