  Cooperative multi-agent reinforcement learning is a decentralized paradigm in
sequential decision making where agents distributed over a network iteratively
collaborate with neighbors to maximize global (network-wide) notions of
rewards. Exact computations typically involve a complexity that scales
exponentially with the number of agents. To address this curse of
dimensionality, we design a scalable algorithm based on the Natural Policy
Gradient framework that uses local information and only requires agents to
communicate with neighbors within a certain range. Under standard assumptions
on the spatial decay of correlations for the transition dynamics of the
underlying Markov process and the localized learning policy, we show that our
algorithm converges to the globally optimal policy with a dimension-free
statistical and computational complexity, incurring a localization error that
does not depend on the number of agents and converges to zero exponentially
fast as a function of the range of communication.
