  Robotic systems are nowadays capable of solving complex navigation tasks.
However, their capabilities are limited to the knowledge of the designer and
consequently lack generalizability to initially unconsidered situations. This
makes deep reinforcement learning (DRL) especially interesting, as these
algorithms promise a self-learning system only relying on feedback from the
environment. In this paper, we consider the problem of lidar-based robot
navigation in continuous action space using DRL without providing any
goal-oriented or global information. By relying solely on local sensor data to
solve navigation tasks, we design an agent that assigns its own waypoints based
on intrinsic motivation. Our agent is able to learn goal-directed navigation
behavior even when facing only sparse feedback, i.e., delayed rewards when
reaching the target. To address this challenge and the complexity of the
continuous action space, we deploy a hierarchical agent structure in which the
exploration is distributed across multiple layers. Within the hierarchical
structure, our agent self-assigns internal goals and learns to extract
reasonable waypoints to reach the desired target position only based on local
sensor data. In our experiments, we demonstrate the navigation capabilities of
our agent in two environments and show that the hierarchical structure
seriously improves the performance in terms of success rate and success
weighted by path length in comparison to a flat structure. Furthermore, we
provide a real-robot experiment to illustrate that the trained agent can be
easily transferred to a real-world scenario.
