  Improving sample efficiency is a key research problem in reinforcement
learning (RL), and CURL, which uses contrastive learning to extract high-level
features from raw pixels of individual video frames, is an efficient
algorithm~\citep{srinivas2020curl}. We observe that consecutive video frames in
a game are highly correlated but CURL deals with them independently. To further
improve data efficiency, we propose a new algorithm, masked contrastive
representation learning for RL, that takes the correlation among consecutive
inputs into consideration. In addition to the CNN encoder and the policy
network in CURL, our method introduces an auxiliary Transformer module to
leverage the correlations among video frames. During training, we randomly mask
the features of several frames, and use the CNN encoder and Transformer to
reconstruct them based on the context frames. The CNN encoder and Transformer
are jointly trained via contrastive learning where the reconstructed features
should be similar to the ground-truth ones while dissimilar to others. During
inference, the CNN encoder and the policy network are used to take actions, and
the Transformer module is discarded. Our method achieves consistent
improvements over CURL on $14$ out of $16$ environments from DMControl suite
and $21$ out of $26$ environments from Atari 2600 Games. The code is available
at https://github.com/teslacool/m-curl.
