  In lifelong learning, an agent learns throughout its entire life without
resets, in a constantly changing environment, as we humans do. Consequently,
lifelong learning comes with a plethora of research problems such as continual
domain shifts, which result in non-stationary rewards and environment dynamics.
These non-stationarities are difficult to detect and cope with due to their
continuous nature. Therefore, exploration strategies and learning methods are
required that are capable of tracking the steady domain shifts, and adapting to
them. We propose Reactive Exploration to track and react to continual domain
shifts in lifelong reinforcement learning, and to update the policy
correspondingly. To this end, we conduct experiments in order to investigate
different exploration strategies. We empirically show that representatives of
the policy-gradient family are better suited for lifelong learning, as they
adapt more quickly to distribution shifts than Q-learning. Thereby,
policy-gradient methods profit the most from Reactive Exploration and show good
results in lifelong learning with continual domain shifts. Our code is
available at: https://github.com/ml-jku/reactive-exploration.
