  For autonomous vehicles, effective behavior planning is crucial to ensure
safety of the ego car. In many urban scenarios, it is hard to create
sufficiently general heuristic rules, especially for challenging scenarios that
some new human drivers find difficult. In this work, we propose a behavior
planning structure based on reinforcement learning (RL) which is capable of
performing autonomous vehicle behavior planning with a hierarchical structure
in simulated urban environments. Application of the hierarchical structure
allows the various layers of the behavior planning system to be satisfied. Our
algorithms can perform better than heuristic-rule-based methods for elective
decisions such as when to turn left between vehicles approaching from the
opposite direction or possible lane-change when approaching an intersection due
to lane blockage or delay in front of the ego car. Such behavior is hard to
evaluate as correct or incorrect, but for some aggressive expert human drivers
handle such scenarios effectively and quickly. On the other hand, compared to
traditional RL methods, our algorithm is more sample-efficient, due to the use
of a hybrid reward mechanism and heuristic exploration during the training
process. The results also show that the proposed method converges to an optimal
policy faster than traditional RL methods.
