  In this paper, we address the following problem: Given an offline
demonstration dataset from an imperfect expert, what is the best way to
leverage it to bootstrap online learning performance in MDPs. We first propose
an Informed Posterior Sampling-based RL (iPSRL) algorithm that uses the offline
dataset, and information about the expert's behavioral policy used to generate
the offline dataset. Its cumulative Bayesian regret goes down to zero
exponentially fast in N, the offline dataset size if the expert is competent
enough. Since this algorithm is computationally impractical, we then propose
the iRLSVI algorithm that can be seen as a combination of the RLSVI algorithm
for online RL, and imitation learning. Our empirical results show that the
proposed iRLSVI algorithm is able to achieve significant reduction in regret as
compared to two baselines: no offline data, and offline dataset but used
without information about the generative policy. Our algorithm bridges online
RL and imitation learning for the first time.
