  Despite success in many challenging problems, reinforcement learning (RL) is
still confronted with sample inefficiency, which can be mitigated by
introducing prior knowledge to agents. However, many transfer techniques in
reinforcement learning make the limiting assumption that the teacher is an
expert. In this paper, we use the action prior from the Reinforcement Learning
as Inference framework - that is, a distribution over actions at each state
which resembles a teacher policy, rather than a Bayesian prior - to recover
state-of-the-art policy distillation techniques. Then, we propose a class of
adaptive methods that can robustly exploit action priors by combining reward
shaping and auxiliary regularization losses. In contrast to prior work, we
develop algorithms for leveraging suboptimal action priors that may
nevertheless impart valuable knowledge - which we call soft action priors. The
proposed algorithms adapt by adjusting the strength of teacher feedback
according to an estimate of the teacher's usefulness in each state. We perform
tabular experiments, which show that the proposed methods achieve
state-of-the-art performance, surpassing it when learning from suboptimal
priors. Finally, we demonstrate the robustness of the adaptive algorithms in
continuous action deep RL problems, in which adaptive algorithms considerably
improved stability when compared to existing policy distillation methods.
