  Failures are challenging for learning to control physical systems since they
risk damage, time-consuming resets, and often provide little gradient
information. Adding safety constraints to exploration typically requires a lot
of prior knowledge and domain expertise. We present a safety measure which
implicitly captures how the system dynamics relate to a set of failure states.
Not only can this measure be used as a safety function, but also to directly
compute the set of safe state-action pairs. Further, we show a model-free
approach to learn this measure by active sampling using Gaussian processes.
While safety can only be guaranteed after learning the safety measure, we show
that failures can already be greatly reduced by using the estimated measure
during learning.
