  AlphaGo's astonishing performance has ignited an explosive interest in
developing deep reinforcement learning (DRL) for numerous real-world
applications, such as intelligent robotics. However, the often prohibitive
complexity of DRL stands at the odds with the required real-time control and
constrained resources in many DRL applications, limiting the great potential of
DRL powered intelligent devices. While substantial efforts have been devoted to
compressing other deep learning models, existing works barely touch the surface
of compressing DRL. In this work, we first identify that there exists an
optimal model size of DRL that can maximize both the test scores and
efficiency, motivating the need for task-specific DRL agents. We therefore
propose an Auto-Agent-Distiller (A2D) framework, which to our best knowledge is
the first neural architecture search (NAS) applied to DRL to automatically
search for the optimal DRL agents for various tasks that optimize both the test
scores and efficiency. Specifically, we demonstrate that vanilla NAS can easily
fail in searching for the optimal agents, due to its resulting high variance in
DRL training stability, and then develop a novel distillation mechanism to
distill the knowledge from both the teacher agent's actor and critic to
stabilize the searching process and improve the searched agents' optimality.
Extensive experiments and ablation studies consistently validate our findings
and the advantages and general applicability of our A2D, outperforming manually
designed DRL in both the test scores and efficiency. All the codes will be
released upon acceptance.
