  Reinforcement learning (RL) algorithms have demonstrated promising results on
complex tasks, yet often require impractical numbers of samples since they
learn from scratch. Meta-RL aims to address this challenge by leveraging
experience from previous tasks so as to more quickly solve new tasks. However,
in practice, these algorithms generally also require large amounts of on-policy
experience during the meta-training process, making them impractical for use in
many problems. To this end, we propose to learn a reinforcement learning
procedure in a federated way, where individual off-policy learners can solve
the individual meta-training tasks, and then consolidate these solutions into a
single meta-learner. Since the central meta-learner learns by imitating the
solutions to the individual tasks, it can accommodate either the standard
meta-RL problem setting or a hybrid setting where some or all tasks are
provided with example demonstrations. The former results in an approach that
can leverage policies learned for previous tasks without significant amounts of
on-policy data during meta-training, whereas the latter is particularly useful
in cases where demonstrations are easy for a person to provide. Across a number
of continuous control meta-RL problems, we demonstrate significant improvements
in meta-RL sample efficiency in comparison to prior work as well as the ability
to scale to domains with visual observations.
