Abstract: Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the environment: how dynamic agents behave, and how the actions of each of the agents affect the behavior of others. In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We focus on graphics games as a proxy of the real environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by ingesting screenplay and keyboard actions during training. Given a key pressed by the agent, GameGAN "renders" the next screen using a carefully designed generative adversarial network. Our approach offers key advantages over existing work: we design a memory module that builds an internal map of the environment, allowing for the agent to return to previously visited locations with high visual consistency. In addition, GameGAN is able to disentangle static and dynamic components within an image making the behavior of the model more interpretable, and relevant for downstream tasks that require explicit reasoning over dynamic elements. This enables many interesting applications such as swapping different components of the game to build new games that do not exist.





  • If you found this codebase useful in your research, please cite:
  title={Learning to Simulate Dynamic Environments with GameGAN},
  author={Kim, Seung Wook and Zhou, Yuhao and Philion, Jonah and Torralba, Antonio and Fidler, Sanja},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},

Environment Setup

This codebase is tested with Ubuntu 18.04 and python3.8.3

  • Clone the repository
git clone
cd GameGAN_code
  • Install dependencies
pip install -r requirements.txt

Dataset extraction

Please clone and follow to install the VizDoom environment.

  • Copy extraction scripts and run
cp data/ DOOMRNN_DIR/
cp data/ DOOMRNN_DIR/
  • Now, extracted data is saved in 'DOOMRNN_DIR/vizdoom_skip3'
cd GameGAN_code/data
python DOOMRNN_DIR/vizdoom_skip3 vizdoom
  • You should now see .npy files extracted in 'data/vizdoom' directory.

For custom datasets, you can construct .npy files that contain a sequence of image and action pairs and define a dataloader similar to 'class vizdoom_dataset'. Please refer to 'data/'.

-- The above repository is deprecated and VizDoom environment might not run correctly in certain systems. In that case, you can use the docker installtaion of and copy the extraction scripts in the docker environment.


We provide training scripts in './scripts'.

  • For training the full GameGAN model, run:
  • For training the GameGAN model without the external memory module, run:


  • You can monitor the training process with tensorboard:
tensorboard --port=PORT --logdir=./results


  • Different environments might need different hyper-parameters. The most important hyper-parameter is 'recon_loss_multiplier' in '', which usually works well with 0.001 ~ 0.05.
  • Environments that do not need long-term consistency usually works better without the external memory module