Rethinking the Implementation Tricks and Monotonicity Constraint

RIIT

Open-source code for Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning. Our goal is to call for a fair comparison of the performance of MARL algorithms.

Code-level Optimizations

There are so many code-level tricks in the Multi-agent Reinforcement Learning (MARL), such as:

• Value function clipping (clip max Q values for QMIX)
• Value Normalization
• Reward scaling
• Orthogonal initialization and layer scaling
• Adam
• learning rate annealing
• Reward Clipping
• Observation Normalization
• Gradient Clipping
• Large Batch Size
• N-step Returns(including GAE($\lambda$) and Q($\lambda$))
• Rollout Process Number
• $\epsilon$-greedy annealing steps
• Death Agent Masking

Related Works

• Implementation Matters in Deep RL: A Case Study on PPO and TRPO
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games

Finetuned-QMIX

Using a few of tricks above (Bold texts), we enabled QMIX to solve almost all of SMAC's scenarios (finetuned QMIX for each scenarios).

Re-Evaluation

Afterwards, we re-evaluate numerous QMIX variants with normalized the tricks (a genaral set of hyperparameters), and find that QMIX achieves the SOTA.

PyMARL

PyMARL is WhiRL's framework for deep multi-agent reinforcement learning and includes implementations of the following algorithms:

Value-based Methods:

• QMIX: QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• VDN: Value-Decomposition Networks For Cooperative Multi-Agent Learning
• IQL: Independent Q-Learning
• QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning
• Qatten: Qatten: A general framework for cooperative multiagent reinforcement learning
• QPLEX: Qplex: Duplex dueling multi-agent q-learning
• WQMIX: Weighted QMIX: Expanding Monotonic Value Function Factorisation

Actor Critic Methods:

• COMA: Counterfactual Multi-Agent Policy Gradients
• VMIX: Value-Decomposition Multi-Agent Actor-Critics
• FacMADDPG: Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control
• LICA: Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning
• DOP: Off-Policy Multi-Agent Decomposed Policy Gradients
• RIIT: Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning

Installation instructions

Install Python packages

# require Anaconda 3 or Miniconda 3
bash install_dependecies.sh

Set up StarCraft II (2.4.10) and SMAC:

bash install_sc2.sh

This will download SC2.4.10 into the 3rdparty folder and copy the maps necessary to run over.

Command Line Tool

Run an experiment

# For SMAC
python3 src/main.py --config=qmix --env-config=sc2 with env_args.map_name=corridor

# For Cooperative Predator-Prey
python3 src/main.py --config=qmix_prey --env-config=stag_hunt with env_args.map_name=stag_hunt

The config files act as defaults for an algorithm or environment.

They are all located in src/config. --config refers to the config files in src/config/algs --env-config refers to the config files in src/config/envs

Run n parallel experiments

# bash run.sh config_name map_name_list (threads_num arg_list gpu_list experinments_num)
bash run.sh qmix corridor 2 epsilon_anneal_time=500000 0,1 5

xxx_list is separated by ,.

All results will be stored in the Results folder and named with map_name.

Kill all training processes

# all python and game processes of current user will quit.
bash clean.sh

Cite

@article{hu2021rethinking,
      title={Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning}, 
      author={Jian Hu and Siyang Jiang and Seth Austin Harding and Haibin Wu and Shih-wei Liao},
      year={2021},
      eprint={2102.03479},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

GitHub

https://github.com/hijkzzz/pymarl2