  Classifiers deployed in high-stakes real-world applications must output
calibrated confidence scores, i.e. their predicted probabilities should reflect
empirical frequencies. Recalibration algorithms can greatly improve a model's
probability estimates; however, existing algorithms are not applicable in
real-world situations where the test data follows a different distribution from
the training data, and privacy preservation is paramount (e.g. protecting
patient records). We introduce a framework that abstracts out the properties of
recalibration problems under differential privacy constraints. This framework
allows us to adapt existing recalibration algorithms to satisfy differential
privacy while remaining effective for domain-shift situations. Guided by our
framework, we also design a novel recalibration algorithm, accuracy temperature
scaling, that outperforms prior work on private datasets. In an extensive
empirical study, we find that our algorithm improves calibration on
domain-shift benchmarks under the constraints of differential privacy. On the
15 highest severity perturbations of the ImageNet-C dataset, our method
achieves a median ECE of 0.029, over 2x better than the next best recalibration
method and almost 5x better than without recalibration.
