  We introduce a framework for calibrating machine learning models so that
their predictions satisfy explicit, finite-sample statistical guarantees. Our
calibration algorithms work with any underlying model and (unknown)
data-generating distribution and do not require model refitting. The framework
addresses, among other examples, false discovery rate control in multi-label
classification, intersection-over-union control in instance segmentation, and
the simultaneous control of the type-1 error of outlier detection and
confidence set coverage in classification or regression. Our main insight is to
reframe the risk-control problem as multiple hypothesis testing, enabling
techniques and mathematical arguments different from those in the previous
literature. We use the framework to provide new calibration methods for several
core machine learning tasks, with detailed worked examples in computer vision
and tabular medical data.
