  Machine learning models are typically deployed in a test setting that differs
from the training setting, potentially leading to decreased model performance
because of domain shift. If we could estimate the performance that a
pre-trained model would achieve on data from a specific deployment setting, for
example a certain clinic, we could judge whether the model could safely be
deployed or if its performance degrades unacceptably on the specific data.
Existing approaches estimate this based on the confidence of predictions made
on unlabeled test data from the deployment's domain. We find existing methods
struggle with data that present class imbalance, because the methods used to
calibrate confidence do not account for bias induced by class imbalance,
consequently failing to estimate class-wise accuracy. Here, we introduce
class-wise calibration within the framework of performance estimation for
imbalanced datasets. Specifically, we derive class-specific modifications of
state-of-the-art confidence-based model evaluation methods including
temperature scaling (TS), difference of confidences (DoC), and average
thresholded confidence (ATC). We also extend the methods to estimate Dice
similarity coefficient (DSC) in image segmentation. We conduct experiments on
four tasks and find the proposed modifications consistently improve the
estimation accuracy for imbalanced datasets. Our methods improve accuracy
estimation by 18\% in classification under natural domain shifts, and double
the estimation accuracy on segmentation tasks, when compared with prior
methods.
