  Clinical machine learning models show a significant performance drop when
tested in settings not seen during training. Domain generalisation models
promise to alleviate this problem, however, there is still scepticism about
whether they improve over traditional training. In this work, we take a
principled approach to identifying Out of Distribution (OoD) environments,
motivated by the problem of cross-hospital generalization in critical care. We
propose model-based and heuristic approaches to identify OoD environments and
systematically compare models with different levels of held-out information. We
find that access to OoD data does not translate to increased performance,
pointing to inherent limitations in defining potential OoD environments
potentially due to data harmonisation and sampling. Echoing similar results
with other popular clinical benchmarks in the literature, new approaches are
required to evaluate robust models on health records.
