  Cooperative Multi-agent Reinforcement Learning (MARL) has attracted
significant attention and played the potential for many real-world
applications. Previous arts mainly focus on facilitating the coordination
ability from different aspects (e.g., non-stationarity, credit assignment) in
single-task or multi-task scenarios, ignoring the stream of tasks that appear
in a continual manner. This ignorance makes the continual coordination an
unexplored territory, neither in problem formulation nor efficient algorithms
designed. Towards tackling the mentioned issue, this paper proposes an approach
Multi-Agent Continual Coordination via Progressive Task Contextualization,
dubbed MACPro. The key point lies in obtaining a factorized policy, using
shared feature extraction layers but separated independent task heads, each
specializing in a specific class of tasks. The task heads can be progressively
expanded based on the learned task contextualization. Moreover, to cater to the
popular CTDE paradigm in MARL, each agent learns to predict and adopt the most
relevant policy head based on local information in a decentralized manner. We
show in multiple multi-agent benchmarks that existing continual learning
methods fail, while MACPro is able to achieve close-to-optimal performance.
More results also disclose the effectiveness of MACPro from multiple aspects
like high generalization ability.
