  Mastering deep reinforcement learning (DRL) proves challenging in tasks
featuring scant rewards. These limited rewards merely signify whether the task
is partially or entirely accomplished, necessitating various exploration
actions before the agent garners meaningful feedback. Consequently, the
majority of existing DRL exploration algorithms struggle to acquire practical
policies within a reasonable timeframe. To address this challenge, we introduce
an approach leveraging offline demonstration trajectories for swifter and more
efficient online RL in environments with sparse rewards. Our pivotal insight
involves treating offline demonstration trajectories as guidance, rather than
mere imitation, allowing our method to learn a policy whose distribution of
state-action visitation marginally matches that of offline demonstrations. We
specifically introduce a novel trajectory distance relying on maximum mean
discrepancy (MMD) and cast policy optimization as a distance-constrained
optimization problem. We then illustrate that this optimization problem can be
streamlined into a policy-gradient algorithm, integrating rewards shaped by
insights from offline demonstrations. The proposed algorithm undergoes
evaluation across extensive discrete and continuous control tasks with sparse
and misleading rewards. The experimental findings demonstrate the significant
superiority of our proposed algorithm over baseline methods concerning diverse
exploration and the acquisition of an optimal policy.
