  We investigate whether self-supervised learning (SSL) can improve online
reinforcement learning (RL) from pixels. We extend the contrastive
reinforcement learning framework (e.g., CURL) that jointly optimizes SSL and RL
losses and conduct an extensive amount of experiments with various
self-supervised losses. Our observations suggest that the existing SSL
framework for RL fails to bring meaningful improvement over the baselines only
taking advantage of image augmentation when the same amount of data and
augmentation is used. We further perform evolutionary searches to find the
optimal combination of multiple self-supervised losses for RL, but find that
even such a loss combination fails to meaningfully outperform the methods that
only utilize carefully designed image augmentations. After evaluating these
approaches together in multiple different environments including a real-world
robot environment, we confirm that no single self-supervised loss or image
augmentation method can dominate all environments and that the current
framework for joint optimization of SSL and RL is limited. Finally, we conduct
the ablation study on multiple factors and demonstrate the properties of
representations learned with different approaches.
