  We introduce robustness in \textit{restless multi-armed bandits} (RMABs), a
popular model for constrained resource allocation among independent stochastic
processes (arms). Nearly all RMAB techniques assume stochastic dynamics are
precisely known. However, in many real-world settings, dynamics are estimated
with significant \emph{uncertainty}, e.g., via historical data, which can lead
to bad outcomes if ignored. To address this, we develop an algorithm to compute
minimax regret -- robust policies for RMABs. Our approach uses a double oracle
framework (oracles for \textit{agent} and \textit{nature}), which is often used
for single-process robust planning but requires significant new techniques to
accommodate the combinatorial nature of RMABs. Specifically, we design a deep
reinforcement learning (RL) algorithm, DDLPO, which tackles the combinatorial
challenge by learning an auxiliary "$\lambda$-network" in tandem with policy
networks per arm, greatly reducing sample complexity, with guarantees on
convergence. DDLPO, of general interest, implements our reward-maximizing agent
oracle. We then tackle the challenging regret-maximizing nature oracle, a
non-stationary RL challenge, by formulating it as a multi-agent RL problem
between a policy optimizer and adversarial nature. This formulation is of
general interest -- we solve it for RMABs by creating a multi-agent extension
of DDLPO with a shared critic. We show our approaches work well in three
experimental domains.
