  Supervised regression to demonstrations has been demonstrated to be a stable
way to train deep policy networks. We are motivated to study how we can take
full advantage of supervised loss functions for stably training deep
reinforcement learning agents. This is a challenging task because it is unclear
how the training data could be collected to enable policy improvement. In this
work, we propose Self-Supervised Reinforcement Learning (SSRL), a simple
algorithm that optimizes policies with purely supervised losses. We demonstrate
that, without policy gradient or value estimation, an iterative procedure of
``labeling" data and supervised regression is sufficient to drive stable policy
improvement. By selecting and imitating trajectories with high episodic
rewards, SSRL is surprisingly competitive to contemporary algorithms with more
stable performance and less running time, showing the potential of solving
reinforcement learning with supervised learning techniques. The code is
available at https://github.com/daochenzha/SSRL
