  With the increasing use of Machine Learning (ML) in critical autonomous
systems, runtime monitors have been developed to detect prediction errors and
keep the system in a safe state during operations. Monitors have been proposed
for different applications involving diverse perception tasks and ML models,
and specific evaluation procedures and metrics are used for different contexts.
This paper introduces three unified safety-oriented metrics, representing the
safety benefits of the monitor (Safety Gain), the remaining safety gaps after
using it (Residual Hazard), and its negative impact on the system's performance
(Availability Cost). To compute these metrics, one requires to define two
return functions, representing how a given ML prediction will impact expected
future rewards and hazards. Three use-cases (classification, drone landing, and
autonomous driving) are used to demonstrate how metrics from the literature can
be expressed in terms of the proposed metrics. Experimental results on these
examples show how different evaluation choices impact the perceived performance
of a monitor. As our formalism requires us to formulate explicit safety
assumptions, it allows us to ensure that the evaluation conducted matches the
high-level system requirements.
