  Automated decision support systems promise to help human experts solve
multiclass classification tasks more efficiently and accurately. However,
existing systems typically require experts to understand when to cede agency to
the system or when to exercise their own agency. Otherwise, the experts may be
better off solving the classification tasks on their own. In this work, we
develop an automated decision support system that, by design, does not require
experts to understand when to trust the system to improve performance. Rather
than providing (single) label predictions and letting experts decide when to
trust these predictions, our system provides sets of label predictions
constructed using conformal prediction$\unicode{x2014}$prediction
sets$\unicode{x2014}$and forcefully asks experts to predict labels from these
sets. By using conformal prediction, our system can precisely trade-off the
probability that the true label is not in the prediction set, which determines
how frequently our system will mislead the experts, and the size of the
prediction set, which determines the difficulty of the classification task the
experts need to solve using our system. In addition, we develop an efficient
and near-optimal search method to find the conformal predictor under which the
experts benefit the most from using our system. Simulation experiments using
synthetic and real expert predictions demonstrate that our system may help
experts make more accurate predictions and is robust to the accuracy of the
classifier the conformal predictor relies on.
