SportsCap
Xin Chen, Anqi Pang, Wei Yang, Yuexin Ma, Lan Xu, Kun Zhou, Jingyi Yu.
This repository contains the official implementation for the paper: SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos (IJCV 2021). Our work is capable of simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input.
Abstract
Markerless motion capture and understanding of professional non-daily human movements is an important yet unsolved task, which suffers from complex motion patterns and severe self-occlusion, especially for the monocular setting. In this paper, we propose SportsCap -- the first approach for simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input. Our approach utilizes the semantic and temporally structured sub-motion prior in the embedding space for motion capture and understanding in a data-driven multi-task manner. Comprehensive experiments on both public and our proposed datasets show that with a challenging monocular sports video input, our novel approach not only significantly improves the accuracy of 3D human motion capture, but also recovers accurate fine-grained semantic action attributes.
The SMART Dataset
SportsCap proposes a challenging sports dataset called Sports Motion and Recognition Tasks (SMART) dataset, which contains per-frame action labels, manually annotated pose, and action assessment of various challenging sports video clips from professional referees.
Download
You can download the SMART dataset (17 GB, version 1.0) from the Google Drive [SMART_part01 | SMART_part02]. The SMART dataset includes source images (>60,000), annotations(>45,000, both pose and action), sport motion embedding spaces, videos (coming soon) and tools.
Annotation
Please load these JSON files in python to parse these annotations about 2D key-points of poses and fine-grained action labels.
Table_VideoInfo_diving.json
Table_VideoInfo_gym.json
Table_VideoInfo_polevalut_highjump_badminton.json
Tools
The tools folder includes several functions to load the annotation and calculate the pose variables. More useful scripts are coming soon.
utils.py - json_load, crop_img_skes, cal_body_bbox ...
Sports Motion Embedding Spaces
With the annotated 2D poses and MoCap 3D pose data, we collect the Sports Motion Embedding Spaces (SMES), the 2D/3D pose priors for various sports. SMES provides strong prior and regularization to ensure that the generated pose result lies in the corresponding action space.
Download
You can download the Motion Embedding Spaces (SMES) (7 MB, version 1.0) separately from GoogleDrive. The released SMES-V1.0 includes many sports, like vault, uneven bar, boxing, diving, hurdles, pole vault, high jump, and so on.
