  Learning new task-specific skills from a few trials is a fundamental
challenge for artificial intelligence. Meta reinforcement learning (meta-RL)
tackles this problem by learning transferable policies that support few-shot
adaptation to unseen tasks. Despite recent advances in meta-RL, most existing
methods require the access to the environmental reward function of new tasks to
infer the task objective, which is not realistic in many practical
applications. To bridge this gap, we study the problem of few-shot adaptation
in the context of human-in-the-loop reinforcement learning. We develop a
meta-RL algorithm that enables fast policy adaptation with preference-based
feedback. The agent can adapt to new tasks by querying human's preference
between behavior trajectories instead of using per-step numeric rewards. By
extending techniques from information theory, our approach can design query
sequences to maximize the information gain from human interactions while
tolerating the inherent error of non-expert human oracle. In experiments, we
extensively evaluate our method, Adaptation with Noisy OracLE (ANOLE), on a
variety of meta-RL benchmark tasks and demonstrate substantial improvement over
baseline algorithms in terms of both feedback efficiency and error tolerance.
