  Vital signs are crucial in intensive care units (ICUs). They are used to
track the patient's state and to identify clinically significant changes.
Predicting vital sign trajectories is valuable for early detection of adverse
events. However, conventional machine learning metrics like RMSE often fail to
capture the true clinical relevance of such predictions. We introduce novel
vital sign prediction performance metrics that align with clinical contexts,
focusing on deviations from clinical norms, overall trends, and trend
deviations. These metrics are derived from empirical utility curves obtained in
a previous study through interviews with ICU clinicians. We validate the
metrics' usefulness using simulated and real clinical datasets (MIMIC and
eICU). Furthermore, we employ these metrics as loss functions for neural
networks, resulting in models that excel in predicting clinically significant
events. This research paves the way for clinically relevant machine learning
model evaluation and optimization, promising to improve ICU patient care. 10
pages, 9 figures.
