  Model-free control based on the idea of Reinforcement Learning is a promising
approach that has recently gained extensive attention. However,
Reinforcement-Learning-based control methods solely focus on the regulation
problem or learn to track a reference that is generated by a time-invariant
exo-system. In the latter case, controllers are only able to track the
time-invariant reference dynamics which they have been trained on and need to
be re-trained each time the reference dynamics change. Consequently, these
methods fail in a number of applications which obviously rely on a trajectory
not being generated by an exo-system. One prominent example is autonomous
driving. This paper provides for the first time an adaptive optimal control
method capable to track reference trajectories not being generated by a
time-invariant exo-system. The main innovation is a novel Q-function that
directly incorporates a given reference trajectory on a moving horizon. This
new Q-function exhibits a particular structure which allows the design of an
efficient, iterative, provably convergent Reinforcement Learning algorithm that
enables optimal tracking. Two real-world examples demonstrate the effectiveness
of our new method.
