  The transfer of reinforcement learning (RL) techniques into real-world
applications is challenged by safety requirements in the presence of physical
limitations. Most RL methods, in particular the most popular algorithms, do not
support explicit consideration of state and input constraints. In this paper,
we address this problem for nonlinear systems with continuous state and input
spaces by introducing a predictive safety filter, which is able to turn a
constrained dynamical system into an unconstrained safe system and to which any
RL algorithm can be applied `out-of-the-box'. The predictive safety filter
receives the proposed control input and decides, based on the current system
state, if it can be safely applied to the real system, or if it has to be
modified otherwise. Safety is thereby established by a continuously updated
safety policy, which is based on a model predictive control formulation using a
data-driven system model and considering state and input dependent
uncertainties.
