<p>Non-binary choices complicate analysis unnecessarily, as most multiple alternative scenarios can be distilled into multiple binary choices.</p>
<p>Outside of metacognitive literature, I have seen studies that assume the naive Bayesian approach for multi-choice decision confidence, for example, from <a href="https://doi.org/10.1016/j.ijforecast.2013.09.009" rel="nofollow noreferrer">Satopää et al (2014)</a> in forecasting:</p>
<blockquote>
<p>For now, assume that the event can take exactly one of a total of M≥2
different outcomes. Under pure ignorance, the forecaster should assign
a probability of 1/M to each outcome. The more ignorant the forecaster
is, the more we would expect him to shrink his forecasts towards 1/M.</p>
</blockquote>
<p>However, this assumption may not hold in the uncalibrated confidence judgments of metacognitive tasks.  There is an interesting paper by <a href="https://doi.org/10.1038/s41467-020-15581-6" rel="nofollow noreferrer">Li &amp; Ma (2020)</a>:</p>
<blockquote>
<p>Experiments on confidence reports have almost exclusively focused on
two-alternative decision-making. ... Here, we test ... a
three-alternative visual categorization task. We found that confidence
reports are best explained by the difference between the posterior
probabilities of the best and the next-best options, rather than by
the posterior probability of the chosen (best) option alone, or by the
overall uncertainty (entropy) of the posterior distribution. Our
results upend the leading notion of decision confidence and instead
suggest that confidence reflects the observer’s subjective probability
that they made the best possible decision.</p>
</blockquote>
<p>This would make even 4-choice tasks much more difficult to interpret.</p>

CC BY-SA 4.0
2021-03-21T05:12:46.867Z
2021-04-06T18:27:14.060Z
2021-04-06T18:27:14.060Z
Arnon Weinberg
<p>In metacognition literature, why is confidence, regardless of scale (Likert-type or continuous) and definition (e.g. decision confidence as a subjective probability of a decision being correct), mostly measured in binary decision tasks, such as 2AFC?</p>
<p>Is it psychologically valid to obtain confidence in other types of choice tasks, such as multiple-alternatives or ordinal ones, in a similar way?</p>

CC BY-SA 4.0
2021-03-20T23:44:25.257Z
Confidence gets complicated when you have more than two choices. If you have options ABCD you may be very confident it is either A or B but not actually certain that A is the correct answer. In this case you have a mixture of confidences at different levels of comparison. Two-alternative tasks are easier to analyze for many reasons and often multiple alternative tasks can be distilled to two choice tests anyways.
Bryan Krause
CC BY-SA 4.0
2021-03-21T01:12:08.633Z
Thank you, Bryan. It’s really an insightful example. But I’m still wondering if people are good at probabilities (Bayesian), an ordinary single-step way for measuring confidence (as probability of choice among ABCD being correct) and a hierarchical way, e.g. first, prob. of AB being correct (e.g. 90% for sure), and then prob of A being correct between A and B (e.g. 50% for sure) make a big difference: i.e. the former may directly yield 45 %, and the latter does 90% x 50% = 45%. It would be appreciated if you or someone else could cite some theories or previous studies if any. Best,
Shelly C.
<measurement><methodology><metacognition><confidence>
How to measure confidence in non-binary (e.g. ordinal) choice tasks?
Shelly C.
# How to measure confidence in non-binary (e.g. ordinal) choice tasks?

(Asked by: username_0 on 2021-03-20)

In metacognition literature, why is confidence, regardless of scale (Likert-type or continuous) and definition (e.g. decision confidence as a subjective probability of a decision being correct), mostly measured in binary decision tasks, such as 2AFC?

Is it psychologically valid to obtain confidence in other types of choice tasks, such as multiple-alternatives or ordinal ones, in a similar way?

**Tags:** measurement, methodology, metacognition, confidence

> **Comments:**
> **username_1**: Confidence gets complicated when you have more than two choices. If you have options ABCD you may be very confident it is either A or B but not actually certain that A is the correct answer. In this case you have a mixture of confidences at different levels of comparison. Two-alternative tasks are easier to analyze for many reasons and often multiple alternative tasks can be distilled to two choice tests anyways.  
> **username_0**: Thank you, Bryan. It’s really an insightful example. But I’m still wondering if people are good at probabilities (Bayesian), an ordinary single-step way for measuring confidence (as probability of choice among ABCD being correct) and a hierarchical way, e.g. first, prob. of AB being correct (e.g. 90% for sure), and then prob of A being correct between A and B (e.g. 50% for sure) make a big difference: i.e. the former may directly yield 45 %, and the latter does 90% x 50% = 45%. It would be appreciated if you or someone else could cite some theories or previous studies if any. Best,  

---

## 1 answers

### Answer by username_2 on 2021-03-21. Score: 0 | Accepted answer

Non-binary choices complicate analysis unnecessarily, as most multiple alternative scenarios can be distilled into multiple binary choices.

Outside of metacognitive literature, I have seen studies that assume the naive Bayesian approach for multi-choice decision confidence, for example, from [Satopää et al (2014)](https://doi.org/10.1016/j.ijforecast.2013.09.009) in forecasting:

> For now, assume that the event can take exactly one of a total of M≥2
> different outcomes. Under pure ignorance, the forecaster should assign
> a probability of 1/M to each outcome. The more ignorant the forecaster
> is, the more we would expect him to shrink his forecasts towards 1/M.

However, this assumption may not hold in the uncalibrated confidence judgments of metacognitive tasks. There is an interesting paper by [Li & Ma (2020)](https://doi.org/10.1038/s41467-020-15581-6):

> Experiments on confidence reports have almost exclusively focused on
> two-alternative decision-making. ... Here, we test ... a
> three-alternative visual categorization task. We found that confidence
> reports are best explained by the difference between the posterior
> probabilities of the best and the next-best options, rather than by
> the posterior probability of the chosen (best) option alone, or by the
> overall uncertainty (entropy) of the posterior distribution. Our
> results upend the leading notion of decision confidence and instead
> suggest that confidence reflects the observer’s subjective probability
> that they made the best possible decision.

This would make even 4-choice tasks much more difficult to interpret.


---

