Categorical DQN.

Implementation of the Categorical DQN as described in A distributional
Perspective on Reinforcement Learning

Thanks to @tudor-berariu for optimisation
and training tricks and for catching two nasty bugs.


You can take a look in the env export file for the full
list of dependencies.

Install the game of Catch:

git clone
cd gym_fast_envs

pip install -r requirements.txt
pip install -e .

Install visdom for reporting: pip install visdom.


First start the visdom server: python -m visdom.server. If you don't want to install or use visdom make sure you deactivate the display_plots option in the configs.

Train the Categorical DQN with python -cf configs/catch_categorical.yaml.

Train a DQN baseline with python -cf configs/catch_dqn.yaml.

To Do

  • [x] Migrate to Pytorch 0.2.0. Breaks compatibility with 0.1.12.
  • [x] Add some training curves.
  • [x] Run on Atari.
  • [x] Add proper evaluation.


First row is with batch size of 64, the second with 32. Will run on more seeds and average for a better comparison. Working on adding Atari results.