  Recent reinforcement learning (RL) methods have achieved success in various
domains. However, multi-agent RL (MARL) remains a challenge in terms of
decentralization, partial observability and scalability to many agents.
Meanwhile, collective behavior requires resolution of the aforementioned
challenges, and remains of importance to many state-of-the-art applications
such as active matter physics, self-organizing systems, opinion dynamics, and
biological or robotic swarms. Here, MARL via mean field control (MFC) offers a
potential solution to scalability, but fails to consider decentralized and
partially observable systems. In this paper, we enable decentralized behavior
of agents under partial information by proposing novel models for decentralized
partially observable MFC (Dec-POMFC), a broad class of problems with
permutation-invariant agents allowing for reduction to tractable single-agent
Markov decision processes (MDP) with single-agent RL solution. We provide
rigorous theoretical results, including a dynamic programming principle,
together with optimality guarantees for Dec-POMFC solutions applied to finite
swarms of interest. Algorithmically, we propose Dec-POMFC-based policy gradient
methods for MARL via centralized training and decentralized execution, together
with policy gradient approximation guarantees. In addition, we improve upon
state-of-the-art histogram-based MFC by kernel methods, which is of separate
interest also for fully observable MFC. We evaluate numerically on
representative collective behavior tasks such as adapted Kuramoto and Vicsek
swarming models, being on par with state-of-the-art MARL. Overall, our
framework takes a step towards RL-based engineering of artificial collective
behavior via MFC.
