Commander-Soldiers Reinforcement Learning for Cooperative Multi-Agent Systems

In ball sports, such as basketball, the coach can guide players to better offend and defend from a holistic perspective to win the game. Inspired by such scenarios, we introduce a coach-like concept into the decision-making process of cooperative multi-agent systems. We propose a new framework Commander-Soldiers Reinforcement Learning (CSRL), for Multi-Agent systems. Specifically, we introduce a virtual role, Commander, which can obtain and encode global information every T steps and send the encoded global guidance to Soldiers (real agents). Furthermore, we propose Policy Guidance Network (PGN), which can customize the encoded global guidance from Commander based on observations for each Soldier, providing each Soldier with specified guidance to the decision-making process. The Soldier takes into account not only the local action-observation histories but also the specified guidance from PGN when making decisions. We validate CSRL on the challenging StarCraft II micromanagement benchmark, proving that our approach can take advantage of intermittent global information to improve collaborative performance.