  Unsupervised visual representation learning offers the opportunity to
leverage large corpora of unlabeled trajectories to form useful visual
representations, which can benefit the training of reinforcement learning (RL)
algorithms. However, evaluating the fitness of such representations requires
training RL algorithms which is computationally intensive and has high variance
outcomes. Inspired by the vision community, we study whether linear probing can
be a proxy evaluation task for the quality of unsupervised RL representation.
Specifically, we probe for the observed reward in a given state and the action
of an expert in a given state, both of which are generally applicable to many
RL domains. Through rigorous experimentation, we show that the probing tasks
are strongly rank correlated with the downstream RL performance on the
Atari100k Benchmark, while having lower variance and up to 600x lower
computational cost. This provides a more efficient method for exploring the
space of pretraining algorithms and identifying promising pretraining recipes
without the need to run RL evaluations for every setting. Leveraging this
framework, we further improve existing self-supervised learning (SSL) recipes
for RL, highlighting the importance of the forward model, the size of the
visual backbone, and the precise formulation of the unsupervised objective.
