Efficient Two-Step Networks for Temporal Action Segmentation

This repository provides a PyTorch implementation of the paper Efficient Two-Step Networks for Temporal Action Segmentation.


* Python 3.8.5
* pyTorch 1.8.1

You can download packages using requirements.txt.
pip install -r requirements.txt


  • Download the data provided by MS-TCN, which contains the I3D features (w/o fine-tune) and the ground truth labels for 3 datasets. (~30GB)
  • Extract it so that you have the data folder in the same directory as train.py.

directory structure

├── config
│   ├── 50salads
│   ├── breakfast
│   └── gtea
├── csv
│   ├── 50salads
│   ├── breakfast
│   └── gtea
├─ dataset ─── 50salads/...
│           ├─ breakfast/...
│           └─ gtea ─── features/
│                    ├─ groundTruth/
│                    ├─ splits/
│                    └─ mapping.txt
├── libs
├── result
├── utils 
├── requirements.txt
├── train.py
├── eval.py
└── README.md

Training and Testing of ETSN


First, convert ground truth files into numpy array.

python utils/generate_gt_array.py ./dataset

Then, please run the below script to generate csv files for data laoder’.

python utils/builda_dataset.py ./dataset


You can train a model by changing the settings of the configuration file.

python train.py ./config/xxx/xxx/config.yaml


You can evaluate the performance of result after running.

python eval.py ./result/xxx/xxx/config.yaml test

We also provide trained ETSN model in Google Drive. Extract it so that you have the result folder in the same directory as train.py.

average cross validation results

python utils/average_cv_results.py [result_dir]


If you find our code useful, please cite our paper.

author = {Yunheng Li and Zhuben Dong and Kaiyuan Liu and Lin Feng and Lianyu Hu and Jie Zhu and Li Xu and Yuhan wang and Shenglan Liu},
journal = {Neurocomputing},
title = {Efficient Two-Step Networks for Temporal Action Segmentation},
year = {2021},
volume = {454},
pages = {373-381},
issn = {0925-2312},
doi = {https://doi.org/10.1016/j.neucom.2021.04.121},
url = {https://www.sciencedirect.com/science/article/pii/S0925231221006998},



For any question, please raise an issue or contact.


We appreciate MS-TCN for extracted I3D feature, backbone network and evaluation code.

Appreciating Yuchi Ishikawa shares the re-implementation of MS-TCN with pytorch.


View Github