  Training deep reinforcement learning (DRL) models usually requires high
computation costs. Therefore, compressing DRL models possesses immense
potential for training acceleration and model deployment. However, existing
methods that generate small models mainly adopt the knowledge
distillation-based approach by iteratively training a dense network. As a
result, the training process still demands massive computing resources. Indeed,
sparse training from scratch in DRL has not been well explored and is
particularly challenging due to non-stationarity in bootstrap training. In this
work, we propose a novel sparse DRL training framework, "the Rigged
Reinforcement Learning Lottery" (RLx2), which builds upon gradient-based
topology evolution and is capable of training a sparse DRL model based entirely
on a sparse network. Specifically, RLx2 introduces a novel multi-step TD target
mechanism with a dynamic-capacity replay buffer to achieve robust value
learning and efficient topology exploration in sparse models. It also reaches
state-of-the-art sparse training performance in several tasks, showing
7.5\times-20\times model compression with less than 3% performance degradation
and up to 20\times and 50\times FLOPs reduction for training and inference,
respectively.
