  Machine Learning~(ML) has provided promising results in recent years across
different applications and domains. However, in many cases, qualities such as
reliability or even safety need to be ensured. To this end, one important
aspect is to determine whether or not ML components are deployed in situations
that are appropriate for their application scope. For components whose
environments are open and variable, for instance those found in autonomous
vehicles, it is therefore important to monitor their operational situation to
determine its distance from the ML components' trained scope. If that distance
is deemed too great, the application may choose to consider the ML component
outcome unreliable and switch to alternatives, e.g. using human operator input
instead. SafeML is a model-agnostic approach for performing such monitoring,
using distance measures based on statistical testing of the training and
operational datasets. Limitations in setting SafeML up properly include the
lack of a systematic approach for determining, for a given application, how
many operational samples are needed to yield reliable distance information as
well as to determine an appropriate distance threshold. In this work, we
address these limitations by providing a practical approach and demonstrate its
use in a well known traffic sign recognition problem, and on an example using
the CARLA open-source automotive simulator.
