  Decision support systems based on prediction sets help humans solve
multiclass classification tasks by narrowing down the set of potential label
values to a subset of them, namely a prediction set, and asking them to always
predict label values from the prediction sets. While this type of systems have
been proven to be effective at improving the average accuracy of the
predictions made by humans, by restricting human agency, they may cause
harm$\unicode{x2014}$a human who has succeeded at predicting the ground-truth
label of an instance on their own may have failed had they used these systems.
In this paper, our goal is to control how frequently a decision support system
based on prediction sets may cause harm, by design. To this end, we start by
characterizing the above notion of harm using the theoretical framework of
structural causal models. Then, we show that, under a natural, albeit
unverifiable, monotonicity assumption, we can estimate how frequently a system
may cause harm using only predictions made by humans on their own. Further, we
also show that, under a weaker monotonicity assumption, which can be verified
experimentally, we can bound how frequently a system may cause harm again using
only predictions made by humans on their own. Building upon these assumptions,
we introduce a computational framework to design decision support systems based
on prediction sets that are guaranteed to cause harm less frequently than a
user-specified value using conformal risk control. We validate our framework
using real human predictions from two different human subject studies and show
that, in decision support systems based on prediction sets, there is a
trade-off between accuracy and counterfactual harm.
