  As deep neural networks are more commonly deployed in high-stakes domains,
their black-box nature makes uncertainty quantification challenging. We
investigate the presentation of conformal prediction sets--a distribution-free
class of methods for generating prediction sets with specified coverage--to
express uncertainty in AI-advised decision-making. Through a large online
experiment, we compare the utility of conformal prediction sets to displays of
Top-1 and Top-k predictions for AI-advised image labeling. In a pre-registered
analysis, we find that the utility of prediction sets for accuracy varies with
the difficulty of the task: while they result in accuracy on par with or less
than Top-1 and Top-k displays for easy images, prediction sets offer some
advantage in assisting humans in labeling out-of-distribution (OOD) images in
the setting that we studied, especially when the set size is small. Our results
empirically pinpoint practical challenges of conformal prediction sets and
provide implications on how to incorporate them for real-world decision-making.
