  Fine-tuning reinforcement learning (RL) models has been challenging because
of a lack of large scale off-the-shelf datasets as well as high variance in
transferability among different environments. Recent work has looked at
tackling offline RL from the perspective of sequence modeling with improved
results as result of the introduction of the Transformer architecture. However,
when the model is trained from scratch, it suffers from slow convergence
speeds. In this paper, we look to take advantage of this formulation of
reinforcement learning as sequence modeling and investigate the transferability
of pre-trained sequence models on other domains (vision, language) when
finetuned on offline RL tasks (control, games). To this end, we also propose
techniques to improve transfer between these domains. Results show consistent
performance gains in terms of both convergence speed and reward on a variety of
environments, accelerating training by 3-6x and achieving state-of-the-art
performance in a variety of tasks using Wikipedia-pretrained and GPT2 language
models. We hope that this work not only brings light to the potentials of
leveraging generic sequence modeling techniques and pre-trained models for RL,
but also inspires future work on sharing knowledge between generative modeling
tasks of completely different domains.
