  We develop a Multi-Agent Reinforcement Learning (MARL) method to learn
scalable control policies for target tracking. Our method can handle an
arbitrary number of pursuers and targets; we show results for tasks consisting
up to 1000 pursuers tracking 1000 targets. We use a decentralized,
partially-observable Markov Decision Process framework to model pursuers as
agents receiving partial observations (range and bearing) about targets which
move using fixed, unknown policies. An attention mechanism is used to
parameterize the value function of the agents; this mechanism allows us to
handle an arbitrary number of targets. Entropy-regularized off-policy RL
methods are used to train a stochastic policy, and we discuss how it enables a
hedging behavior between pursuers that leads to a weak form of cooperation in
spite of completely decentralized control execution. We further develop a
masking heuristic that allows training on smaller problems with few
pursuers-targets and execution on much larger problems. Thorough simulation
experiments, ablation studies, and comparisons to state of the art algorithms
are performed to study the scalability of the approach and robustness of
performance to varying numbers of agents and targets.
