  We propose Value Explicit Pretraining (VEP), a method that learns
generalizable representations for transfer reinforcement learning. VEP enables
learning of new tasks that share similar objectives as previously learned
tasks, by learning an encoder for objective-conditioned representations,
irrespective of appearance changes and environment dynamics. To pre-train the
encoder from a sequence of observations, we use a self-supervised contrastive
loss that results in learning temporally smooth representations. VEP learns to
relate states across different tasks based on the Bellman return estimate that
is reflective of task progress. Experiments using a realistic navigation
simulator and Atari benchmark show that the pretrained encoder produced by our
method outperforms current SoTA pretraining methods on the ability to
generalize to unseen tasks. VEP achieves up to a 2 times improvement in rewards
on Atari and visual navigation, and up to a 3 times improvement in sample
efficiency. For videos of policy performance visit our
https://sites.google.com/view/value-explicit-pretraining/
