  Meta-learning is a line of research that develops the ability to leverage
past experiences to efficiently solve new learning problems. Meta-Reinforcement
Learning (meta-RL) methods demonstrate a capability to learn behaviors that
efficiently acquire and exploit information in several meta-RL problems.
  In this context, the Alchemy benchmark has been proposed by Wang et al.
[2021]. Alchemy features a rich structured latent space that is challenging for
state-of-the-art model-free RL methods. These methods fail to learn to properly
explore then exploit.
  We develop a model-based algorithm. We train a model whose principal block is
a Transformer Encoder to fit the symbolic Alchemy environment dynamics. Then we
define an online planner with the learned model using a tree search method.
This algorithm significantly outperforms previously applied model-free RL
methods on the symbolic Alchemy problem.
  Our results reveal the relevance of model-based approaches with online
planning to perform exploration and exploitation successfully in meta-RL.
Moreover, we show the efficiency of the Transformer architecture to learn
complex dynamics that arise from latent spaces present in meta-RL problems.
