  In many real-world settings, an agent must learn to act in environments where
no reward signal can be specified, but a set of expert demonstrations is
available. Imitation learning (IL) is a popular framework for learning policies
from such demonstrations. However, in some cases, differences in observability
between the expert and the agent can give rise to an imitation gap such that
the expert's policy is not optimal for the agent and a naive application of IL
can fail catastrophically. In particular, if the expert observes the Markov
state and the agent does not, then the expert will not demonstrate the
information-gathering behavior needed by the agent but not the expert. In this
paper, we propose a Bayesian solution to the Imitation Gap (BIG), first using
the expert demonstrations, together with a prior specifying the cost of
exploratory behavior that is not demonstrated, to infer a posterior over
rewards with Bayesian inverse reinforcement learning (IRL). BIG then uses the
reward posterior to learn a Bayes-optimal policy. Our experiments show that
BIG, unlike IL, allows the agent to explore at test time when presented with an
imitation gap, whilst still learning to behave optimally using expert
demonstrations when no such gap exists.
