  Deep reinforcement learning (DRL) is a promising way to achieve human-like
autonomous driving. However, the low sample efficiency and difficulty of
designing reward functions for DRL would hinder its applications in practice.
In light of this, this paper proposes a novel framework to incorporate human
prior knowledge in DRL, in order to improve the sample efficiency and save the
effort of designing sophisticated reward functions. Our framework consists of
three ingredients, namely expert demonstration, policy derivation, and
reinforcement learning. In the expert demonstration step, a human expert
demonstrates their execution of the task, and their behaviors are stored as
state-action pairs. In the policy derivation step, the imitative expert policy
is derived using behavioral cloning and uncertainty estimation relying on the
demonstration data. In the reinforcement learning step, the imitative expert
policy is utilized to guide the learning of the DRL agent by regularizing the
KL divergence between the DRL agent's policy and the imitative expert policy.
To validate the proposed method in autonomous driving applications, two
simulated urban driving scenarios (unprotected left turn and roundabout) are
designed. The strengths of our proposed method are manifested by the training
results as our method can not only achieve the best performance but also
significantly improve the sample efficiency in comparison with the baseline
algorithms (particularly 60\% improvement compared to soft actor-critic). In
testing conditions, the agent trained by our method obtains the highest success
rate and shows diverse and human-like driving behaviors as demonstrated by the
human expert.
