3D position tracking for soccer players with multi-camera videos

3D Player Tracking with Multi-View Stream

Project for 3DV 2021 Spring @ ETH Zurich [Report Link]

This repo contains a full pipeline to support 3D position tracking of soccer players, with multi-view calibrated moving/fixed video sequences as inputs.

In single-camera tracking stage, Tracktor++ is used to get 2D positions.
In multi-camera tracking stage, 2D positions are projected into 3D positions. Then across-camera association is achieved as an optimization problem with spatial, temporal and visual constraints.
In the end, visualization in 2D, 3D and a voronoi visualization for sports coaching purpose are provided.

3D Tracking	Sports Coaching

Demo

Check demo scripts as examples

Currently, processed data is under protection due to legal issues.

Run the demo visualization on the moving cameras

bash script/demo_moving.sh

Run the demo visualization on the fixed cameras

bash script/demo_fix.sh

Preprocessing

Split video into image frames

python src/utils/v2img.py --pathIn=data/0125-0135/CAM1/CAM1.mp4 --pathOut=data/0125-0135/CAM1/img --splitnum=1

Estimate football pitch homography (size 120m * 90m ref)

FIFA official document

python src/utils/computeHomo.py --img=data/0125-0135/RIGHT/img/image0000.jpg --out_dir=data/0125-0135/RIGHT/

Handle moving cameras

python src/utils/mov2static.py --calib_file=data/calibration_results/0125-0135/CAM1/calib.txt --img_dir=data/0125-0135/CAM1/img --output_dir=data/0125-0135/CAM1/img_static

Convert ground truth/annotation json to text file

python src/utils/json2txt.py --jsonfile=data/0125-0135/0125-0135.json

Single-camera tracking

Object Detector: frcnn_fpn

Train object detector and generate detection results with this Google Colab notebook. [pretrained model]
Run Tracktor++

Put trainded object detector model_epoch_50.model into src/tracking_wo_bnw/output/faster_rcnn_fpn_training_soccer/.

Put data and calibration results into src/tracking_wo_bnw/.

cd src/tracking_wo_bnw
python experiments/scripts/test_tracktor.py

Run ReID(team id) model

python src/team_classification/team_svm.py PATH_TO_TRACKING_RESULT PATH_TO_IMAGES

Convert tracking results to coordinates on the pitch

Equation to find the intersection of a line with a plane (ref)

python src/calib.py --calib_path=PATH_TO_CALIB --res_path=PATH_TO_TRACKING_RESULT --xymode --reid

# also plot the camera positions for fixed cameras
python src/calib.py --calib_path=PATH_TO_CALIB --res_path=PATH_TO_TRACKING_RESULT --viz

Across-camera association

Run two-cam tracker

python src/runMCTRacker.py 

# add team id constraint
python src/runMCTRacker.py --doreid

Run multi-cam tracker (e.g. 8 cams)

python src/runTreeMCTracker.py --doreid

Evaluation

Produce quatitative results (visualize results)

visualize 2d bounding box

# if format <x, y, w, h>
python src/utils/visualize.py --img_dir=data/0125-0135/RIGHT/img --result_file=output/tracktor/16m_right_prediction.txt 
# if format <x1, y1, x2, y2>
python src/utils/visualize.py --img_dir=data/0125-0135/RIGHT/img --result_file=output/iou/16m_right.txt --xymode
# if with team id
python src/utils/visualize.py --img_dir=data/0125-0135/RIGHT/img --result_file=output/tracktor/16m_right_prediction.txt --reid
# if 3d mode
python src/utils/visualize.py --img_dir=data/0125-0135/RIGHT/img --result_file=output/tracktor/RIGHT.txt --calib_file=data/calibration_results/0125-0135/RIGHT/calib.txt  --pitchmode

visualize 3d tracking result with ground truth and voronoi diagram

python src/utils/visualize_on_pitch.py --result_file=PATH_TO_TRACKING_RESULT --ground_truth=PATH_TO_GROUND_TRUTH

visualize 3d ground truth on camera frames (reprojection)

python src/utils/visualize_tracab --img_path=PATH_TO_IMAGES --calib_path=PATH_TO_CALIB --gt_path=PATH_TO_TRACAB_GT --output_path=PATH_TO_OUTPUT_VIDEO

Produce quantitative result

# 2d <frame id, objid, x, y, w, h, .., ...>
python src/motmetrics/apps/eval_motchallenge.py data/0125-0135/ output/tracktor_filtered

# 3d
python src/utils/eval3d.py --pred=output/pitch/EPTS_3_pitch.txt_EPTS_4_pitch.txt.txt --fixcam  --gt=data/fixedcam/gt_pitch_550.txt
python src/utils/eval3d.py --fixcam --boxplot