  Learning from visual observation (LfVO), aiming at recovering policies from
only visual observation data, is promising yet a challenging problem. Existing
LfVO approaches either only adopt inefficient online learning schemes or
require additional task-specific information like goal states, making them not
suited for open-ended tasks. To address these issues, we propose a two-stage
framework for learning from visual observation. In the first stage, we
introduce and pretrain State-to-Go (STG) Transformer offline to predict and
differentiate latent transitions of demonstrations. Subsequently, in the second
stage, the STG Transformer provides intrinsic rewards for downstream
reinforcement learning tasks where an agent learns merely from intrinsic
rewards. Empirical results on Atari and Minecraft show that our proposed method
outperforms baselines and in some tasks even achieves performance comparable to
the policy learned from environmental rewards. These results shed light on the
potential of utilizing video-only data to solve difficult visual reinforcement
learning tasks rather than relying on complete offline datasets containing
states, actions, and rewards. The project's website and code can be found at
https://sites.google.com/view/stgtransformer.
