  Meta reinforcement learning (meta-RL) aims to learn a policy solving a set of
training tasks simultaneously and quickly adapting to new tasks. It requires
massive amounts of data drawn from training tasks to infer the common structure
shared among tasks. Without heavy reward engineering, the sparse rewards in
long-horizon tasks exacerbate the problem of sample efficiency in meta-RL.
Another challenge in meta-RL is the discrepancy of difficulty level among
tasks, which might cause one easy task dominating learning of the shared policy
and thus preclude policy adaptation to new tasks. This work introduces a novel
objective function to learn an action translator among training tasks. We
theoretically verify that the value of the transferred policy with the action
translator can be close to the value of the source policy and our objective
function (approximately) upper bounds the value difference. We propose to
combine the action translator with context-based meta-RL algorithms for better
data collection and more efficient exploration during meta-training. Our
approach empirically improves the sample efficiency and performance of meta-RL
algorithms on sparse-reward tasks.
