  Despite advances in hierarchical reinforcement learning, its applications to
path planning in autonomous driving on highways are challenging. One reason is
that conventional hierarchical reinforcement learning approaches are not
amenable to autonomous driving due to its riskiness: the agent must move
avoiding multiple obstacles such as other agents that are highly unpredictable,
thus safe regions are small, scattered, and changeable over time. To overcome
this challenge, we propose a spatially hierarchical reinforcement learning
method for state space and policy space. The high-level policy selects not only
behavioral sub-policy but also regions to pay mind to in state space and for
outline in policy space. Subsequently, the low-level policy elaborates the
short-term goal position of the agent within the outline of the region selected
by the high-level command. The network structure and optimization suggested in
our method are as concise as those of single-level methods. Experiments on the
environment with various shapes of roads showed that our method finds the
nearly optimal policies from early episodes, outperforming a baseline
hierarchical reinforcement learning method, especially in narrow and complex
roads. The resulting trajectories on the roads were similar to those of human
strategies on the behavioral planning level.
