  When groups of people are tasked with making a judgment, the issue of
uncertainty often arises. Existing methods to reduce uncertainty typically
focus on iteratively improving specificity in the overall task instruction.
However, uncertainty can arise from multiple sources, such as ambiguity of the
item being judged due to limited context, or disagreements among the
participants due to different perspectives and an under-specified task. A
one-size-fits-all intervention may be ineffective if it is not targeted to the
right source of uncertainty. In this paper we introduce a new workflow,
Judgment Sieve, to reduce uncertainty in tasks involving group judgment in a
targeted manner. By utilizing measurements that separate different sources of
uncertainty during an initial round of judgment elicitation, we can then select
a targeted intervention adding context or deliberation to most effectively
reduce uncertainty on each item being judged. We test our approach on two
tasks: rating word pair similarity and toxicity of online comments, showing
that targeted interventions reduced uncertainty for the most uncertain cases.
In the top 10% of cases, we saw an ambiguity reduction of 21.4% and 25.7%, and
a disagreement reduction of 22.2% and 11.2% for the two tasks respectively. We
also found through a simulation that our targeted approach reduced the average
uncertainty scores for both sources of uncertainty as opposed to uniform
approaches where reductions in average uncertainty from one source came with an
increase for the other.
