Python 2.7 Python 3.6

Dancing to Music

PyTorch implementation of the cross-modality generative model that synthesizes dance from music.


Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, Jan Kautz
Dancing to Music Neural Information Processing Systems (NeurIPS) 2019
[Paper] [YouTube] [Project] [Blog] [Supp]

Example Videos

  • Beat-Matching
    1st row: generated dance sequences, 2nd row: music beats, 3rd row: kinematics beats

  • Multimodality
    Generate various dance sequences with the same music and the same initial pose.

  • Long-Term Generation
    Seamlessly generate a dance sequence with arbitrary length.

  • Photo-Realisitc Videos
    Map generated dance sequences to photo-realistic videos.

Train Decomposition

python --name Decomp

Train Composition

python --name Decomp --decomp_snapshot DECOMP_SNAPSHOT


python --decomp_snapshot DECOMP_SNAPSHOT --comp_snapshot COMP_SNAPSHOT --aud_path AUD_PATH --out_file OUT_FILE --out_dir OUT_DIR --thr THR
  • Flags

    • aud_path: input .wav file
    • out_file: location of output .mp4 file
    • out_dir: directory of output frames
    • thr: threshold based on motion magnitude
    • modulate: whether to do beat warping
  • Example

python -decomp_snapshot snapshot/Stage1.ckpt --comp_snapshot snapshot/Stage2.ckpt --aud_path demo/demo.wav --out_file demo/out.mp4 --out_dir demo/out_frame


If you find this code useful for your research, please cite our paper:

  title={Dancing to Music},
  author={Lee, Hsin-Ying and Yang, Xiaodong and Liu, Ming-Yu and Wang, Ting-Chun and Lu, Yu-Ding and Yang, Ming-Hsuan and Kautz, Jan},


Copyright (C) 2020 NVIDIA Corporation. All rights reserved. This work is made available under NVIDIA Source Code License (1-Way Commercial). To view a copy of this license, visit