  Calibration is a well-studied property of predictors which guarantees
meaningful uncertainty estimates. Multicalibration is a related notion --
originating in algorithmic fairness -- which requires predictors to be
simultaneously calibrated over a potentially complex and overlapping collection
of protected subpopulations (such as groups defined by ethnicity, race, or
income). We conduct the first comprehensive study evaluating the usefulness of
multicalibration post-processing across a broad set of tabular, image, and
language datasets for models spanning from simple decision trees to 90 million
parameter fine-tuned LLMs. Our findings can be summarized as follows: (1)
models which are calibrated out of the box tend to be relatively
multicalibrated without any additional post-processing; (2) multicalibration
post-processing can help inherently uncalibrated models; and (3) traditional
calibration measures may sometimes provide multicalibration implicitly. More
generally, we also distill many independent observations which may be useful
for practical and effective applications of multicalibration post-processing in
real-world contexts.
