  In this paper, we study Reinforcement Learning from Demonstrations (RLfD)
that improves the exploration efficiency of Reinforcement Learning (RL) by
providing expert demonstrations. Most of existing RLfD methods require
demonstrations to be perfect and sufficient, which yet is unrealistic to meet
in practice. To work on imperfect demonstrations, we first define an imperfect
expert setting for RLfD in a formal way, and then point out that previous
methods suffer from two issues in terms of optimality and convergence,
respectively. Upon the theoretical findings we have derived, we tackle these
two issues by regarding the expert guidance as a soft constraint on regulating
the policy exploration of the agent, which eventually leads to a constrained
optimization problem. We further demonstrate that such problem is able to be
addressed efficiently by performing a local linear search on its dual form.
Considerable empirical evaluations on a comprehensive collection of benchmarks
indicate our method attains consistent improvement over other RLfD
counterparts.
