  Machine learning models are increasingly used in practice. However, many
machine learning methods are sensitive to test or operational data that is
dissimilar to training data. Out-of-distribution (OOD) data is known to
increase the probability of error and research into metrics that identify what
dissimilarities in data affect model performance is on-going. Recently,
combinatorial coverage metrics have been explored in the literature as an
alternative to distribution-based metrics. Results show that coverage metrics
can correlate with classification error. However, other results show that the
utility of coverage metrics is highly dataset-dependent. In this paper, we show
that this dataset-dependence can be alleviated with metric learning, a machine
learning technique for learning latent spaces where data from different classes
is further apart. In a study of 6 open-source datasets, we find that metric
learning increased the difference between set-difference coverage metrics
(SDCCMs) calculated on correctly and incorrectly classified data, thereby
demonstrating that metric learning improves the ability of SDCCMs to anticipate
classification error. Paired t-tests validate the statistical significance of
our findings. Overall, we conclude that metric learning improves the ability of
coverage metrics to anticipate classifier error and identify when OOD data is
likely to degrade model performance.
