  We present Distributed Deep Deterministic Policy Gradient (3DPG), a
multi-agent actor-critic (MAAC) algorithm for Markov games. Unlike previous
MAAC algorithms, 3DPG is fully distributed during both training and deployment.
3DPG agents calculate local policy gradients based on the most recently
available local data (states, actions) and local policies of other agents.
During training, this information is exchanged using a potentially lossy and
delaying communication network. The network therefore induces Age of
Information (AoI) for data and policies. We prove the asymptotic convergence of
3DPG even in the presence of potentially unbounded Age of Information (AoI).
This provides an important step towards practical online and distributed
multi-agent learning since 3DPG does not assume information to be available
deterministically. We analyze 3DPG in the presence of policy and data transfer
under mild practical assumptions. Our analysis shows that 3DPG agents converge
to a local Nash equilibrium of Markov games in terms of utility functions
expressed as the expected value of the agents local approximate action-value
functions (Q-functions). The expectations of the local Q-functions are with
respect to limiting distributions over the global state-action space shaped by
the agents' accumulated local experiences. Our results also shed light on the
policies obtained by general MAAC algorithms. We show through a heuristic
argument and numerical experiments that 3DPG improves convergence over previous
MAAC algorithms that use old actions instead of old policies during training.
Further, we show that 3DPG is robust to AoI; it learns competitive policies
even with large AoI and low data availability.
