  This paper investigates how to incorporate expert observations (without
explicit information on expert actions) into a deep reinforcement learning
setting to improve sample efficiency. First, we formulate an augmented policy
loss combining a maximum entropy reinforcement learning objective with a
behavioral cloning loss that leverages a forward dynamics model. Then, we
propose an algorithm that automatically adjusts the weights of each component
in the augmented loss function. Experiments on a variety of continuous control
tasks demonstrate that the proposed algorithm outperforms various benchmarks by
effectively utilizing available expert observations.
