  In view of its power in extracting feature representation, contrastive
self-supervised learning has been successfully integrated into the practice of
(deep) reinforcement learning (RL), leading to efficient policy learning in
various applications. Despite its tremendous empirical successes, the
understanding of contrastive learning for RL remains elusive. To narrow such a
gap, we study how RL can be empowered by contrastive learning in a class of
Markov decision processes (MDPs) and Markov games (MGs) with low-rank
transitions. For both models, we propose to extract the correct feature
representations of the low-rank model by minimizing a contrastive loss.
Moreover, under the online setting, we propose novel upper confidence bound
(UCB)-type algorithms that incorporate such a contrastive loss with online RL
algorithms for MDPs or MGs. We further theoretically prove that our algorithm
recovers the true representations and simultaneously achieves sample efficiency
in learning the optimal policy and Nash equilibrium in MDPs and MGs. We also
provide empirical studies to demonstrate the efficacy of the UCB-based
contrastive learning method for RL. To the best of our knowledge, we provide
the first provably efficient online RL algorithm that incorporates contrastive
learning for representation learning. Our codes are available at
https://github.com/Baichenjia/Contrastive-UCB.
