  Most multi-agent reinforcement learning approaches adopt two types of policy
optimization methods that either update policy simultaneously or sequentially.
Simultaneously updating policies of all agents introduces non-stationarity
problem. Although sequentially updating policies agent-by-agent in an
appropriate order improves policy performance, it is prone to low efficiency
due to sequential execution, resulting in longer model training and execution
time. Intuitively, partitioning policies of all agents according to their
interdependence and updating joint policy batch-by-batch can effectively
balance performance and efficiency. However, how to determine the optimal batch
partition of policies and batch updating order are challenging problems.
Firstly, a sequential batched policy updating scheme, B2MAPO (Batch by Batch
Multi-Agent Policy Optimization), is proposed with a theoretical guarantee of
the monotonic incrementally tightened bound. Secondly, a universal modulized
plug-and-play B2MAPO hierarchical framework, which satisfies CTDE principle, is
designed to conveniently integrate any MARL models to fully exploit and merge
their merits, including policy optimality and inference efficiency. Next, a
DAG-based B2MAPO algorithm is devised, which is a carefully designed
implementation of B2MAPO framework. Comprehensive experimental results
conducted on StarCraftII Multi-agent Challenge and Google Football Research
demonstrate the performance of DAG-based B2MAPO algorithm outperforms baseline
methods. Meanwhile, compared with A2PO, our algorithm reduces the model
training and execution time by 60.4% and 78.7%, respectively.
