<p>Bias can be quantified in many different ways. In human memory research carried out in the cognitive psychology tradition, there are simple ways to think about it. One basic measure of cognitive bias is merely called <em>bias</em>, and it's a measure of the absolute accuracy of an individual's probability judgments. You average probability judgments across a given subject, then average performance, and subtract one from the other. The magnitude and direction of this discrepancy could be argued to reflect a task-specific cognitive bias.</p>

<p>That's not useful without a concrete example. Say I ask you 10 questions tapping your general knowledge about the world (e.g., "Who invaded Rome by crossing the Alps with elephants?") After you respond, I ask you to rate your confidence on a scale from 0% (not at all confident) to 100% (entirely confident). We do this for nine more questions.</p>

<p>After you've taken this short test, I can easily calculate your bias by averaging your accuracy across the 10 questions (say you get 2 wrong, so 80%) and subtracting your average confidence across those questions (say it's 73%).  It's a 7% difference, and you were more accurate than you were confident, so your bias is said to be 7% -- you were <em>underconfident</em>.</p>

<p>There are lots of other ways you can think about this. Say, for instance, you weren't interested in overconfidence or underconfidence, but rather, just the degree to which your performance differed from your estimations. Here you could just use the squared value of the difference to represent probability accuracy.</p>

CC BY-SA 3.0
2012-03-29T17:22:05.823Z
Thanx, Andy. Just what I was asking for. Is there a general standard, or something almost everybody in the field uses?
Benny Skogberg
CC BY-SA 3.0
2012-03-29T19:25:22.637Z
Unfortunately I'm afraid it's really going to vary from discipline to discipline. For basic research, this was the most atheoretical topic I could come up with. I knew bias and similar measures are used whenever numerically-based probability judgments come into play (in the field of personality psychology, for instance). It's tougher for more qualitative areas, though.
Andy DeSoto
CC BY-SA 3.0
2012-03-29T15:36:37.263Z
2012-03-29T15:36:37.263Z
Andy DeSoto
<p>If you come to this question from the <strong>bayesian tradition</strong>, then there is only one place where you can sneak in bias: your prior. This dovetails nicely with the wikipedia definition:</p>
<blockquote>
<p>a pattern of deviation in judgment that occurs in particular situations, leading to perceptual distortion, inaccurate judgment, illogical interpretation, or what is broadly called irrationality.</p>
</blockquote>
<p>Since bayesian updating is considered to be 'rational', the only place to sneak in 'irrationality' (in quotes because we are using these terms very loosely) is in the prior before you were given any evidence/observations. So for a bayesian, <strong>measuring bias is measuring the prior</strong>.</p>
<h3>Measuring a prior</h3>
<p>Conveniently,  Kalish et al. (2007) have a nice mechanism for measuring people's priors: have <span class="math-container">$n$</span> participants: <span class="math-container">$1, ... , n$</span> and give the first one some real input-output pairs on the relevant task to learn from. To train the <span class="math-container">$i + 1$</span>th participant:</p>
<ol>
<li>take the <span class="math-container">$i$</span>th participant,</li>
<li>give them some inputs and ask them what they think the output should be,</li>
<li>use the input-output pairs they generate to train the <span class="math-container">$i+1$</span>th participant.</li>
</ol>
<p>Then, towards the end of the chain, the participants will start to converge towards their prior or inherent bias.</p>
<h3>Example</h3>
<p>A real example is of people's bias in functional relationships. The first person is given 25 <span class="math-container">$(x,y)$</span> pairs from some function <span class="math-container">$y = f(x)$</span>. A person at stage <span class="math-container">$i$</span> is given 25 <span class="math-container">$(x,y)$</span> pairs from the person at stage <span class="math-container">$i - 1$</span>. The person is then tested by being given an <span class="math-container">$x$</span> value and asked for a <span class="math-container">$y$</span>, 25 times. The results of this are passed on to the person at trial <span class="math-container">$i + 1$</span> as the training data. The results:</p>
<p><img src="https://i.sstatic.net/uf0c5.png" alt="Figure 3 from Kalish et al. (2007)" /></p>
<p>Show that people have a strong bias towards positive linear relationships. One could apply a <a href="http://cocosci.berkeley.edu/publications.php?topic=Cultural%20Evolution%20and%20Iterated%20Learning" rel="noreferrer">similar procedure to other domains</a>, but of course finding a good way to do so is a particular domain is an important research question for an experimental psychologist.</p>
<h3>References</h3>
<ul>
<li>Kalish, M., Griffiths, T. &amp; Lewandowsky, S. (2007). Iterated learning: Intergenerational knowledge transmission reveals inductive biases. <em>Psychonomic Bulletin &amp; Review</em>, 14, 288-294. doi: <a href="http://dx.doi.org/10.3758/BF03194066" rel="noreferrer">10.3758/BF03194066</a></li>
</ul>

CC BY-SA 3.0
2012-05-28T19:02:45.367Z
2012-09-20T11:27:35.173Z
2020-06-17T09:55:31.633Z
Artem Kaznatcheev
<p>We know that bias exists, and that it affects our judgment and perception. This effect has to do with user's experience in life. That experience is taken care of by the brain, and if you counter a situation again, you have a predefined pattern of how to react. This can be changed over time, if other experiences is added. Wikipedia defines <a href="http://en.wikipedia.org/wiki/Cognitive_bias">cognitive bias</a> as: </p>

<blockquote>
  <p>a pattern of deviation in judgment that occurs in particular situations, leading to perceptual distortion, inaccurate judgment, illogical interpretation, or what is broadly called irrationality. Implicit in the concept of a "pattern of deviation" is a standard of comparison with what is normatively expected. </p>
</blockquote>

<p>In other disciplines, measurement is a valuable tool. One would wish for a scale of deviation from "what is normally expected". </p>

<p><strong>Is it possible to quantify cognitive bias?</strong></p>

CC BY-SA 3.0
2012-03-30T18:06:41.623Z
Harvard is doing something with implicit preferences that might interest you: https://implicit.harvard.edu/implicit/demo/selectatest.html
Nate Glenn
CC BY-SA 3.0
2012-03-31T05:17:40.397Z
@NateGlenn Thanx, looks very interesting. I will follow the progress on that research.
Benny Skogberg
<methodology><measurement><bias>
Is it possible to quantify cognitive bias?
Benny Skogberg
# Is it possible to quantify cognitive bias?

(Asked by: username_0 on 2012-03-29)

We know that bias exists, and that it affects our judgment and perception. This effect has to do with user's experience in life. That experience is taken care of by the brain, and if you counter a situation again, you have a predefined pattern of how to react. This can be changed over time, if other experiences is added. Wikipedia defines [cognitive bias](http://en.wikipedia.org/wiki/Cognitive_bias) as:

> a pattern of deviation in judgment that occurs in particular situations, leading to perceptual distortion, inaccurate judgment, illogical interpretation, or what is broadly called irrationality. Implicit in the concept of a "pattern of deviation" is a standard of comparison with what is normatively expected.

In other disciplines, measurement is a valuable tool. One would wish for a scale of deviation from "what is normally expected".

**Is it possible to quantify cognitive bias?**

**Tags:** methodology, measurement, bias

> **Comments:**
> **username_1**: Harvard is doing something with implicit preferences that might interest you: https://implicit.harvard.edu/implicit/demo/selectatest.html  
> **username_0**: @NateGlenn Thanx, looks very interesting. I will follow the progress on that research.  

---

## 2 answers

### Answer by username_3 on 2012-05-28. Score: 9 | Accepted answer

If you come to this question from the **bayesian tradition**, then there is only one place where you can sneak in bias: your prior. This dovetails nicely with the wikipedia definition:

> a pattern of deviation in judgment that occurs in particular situations, leading to perceptual distortion, inaccurate judgment, illogical interpretation, or what is broadly called irrationality.

Since bayesian updating is considered to be 'rational', the only place to sneak in 'irrationality' (in quotes because we are using these terms very loosely) is in the prior before you were given any evidence/observations. So for a bayesian, **measuring bias is measuring the prior**.

### Measuring a prior

Conveniently, Kalish et al. (2007) have a nice mechanism for measuring people's priors: have $n$ participants: $1, ... , n$ and give the first one some real input-output pairs on the relevant task to learn from. To train the $i + 1$th participant:

1. take the $i$th participant,
2. give them some inputs and ask them what they think the output should be,
3. use the input-output pairs they generate to train the $i+1$th participant.

Then, towards the end of the chain, the participants will start to converge towards their prior or inherent bias.

### Example

A real example is of people's bias in functional relationships. The first person is given 25 $(x,y)$ pairs from some function $y = f(x)$. A person at stage $i$ is given 25 $(x,y)$ pairs from the person at stage $i - 1$. The person is then tested by being given an $x$ value and asked for a $y$, 25 times. The results of this are passed on to the person at trial $i + 1$ as the training data. The results:

![Figure 3 from Kalish et al. (2007)](https://i.sstatic.net/uf0c5.png)

Show that people have a strong bias towards positive linear relationships. One could apply a [similar procedure to other domains](http://cocosci.berkeley.edu/publications.php?topic=Cultural%20Evolution%20and%20Iterated%20Learning), but of course finding a good way to do so is a particular domain is an important research question for an experimental psychologist.

### References

* Kalish, M., Griffiths, T. & Lewandowsky, S. (2007). Iterated learning: Intergenerational knowledge transmission reveals inductive biases. *Psychonomic Bulletin & Review*, 14, 288-294. doi: [10.3758/BF03194066](http://dx.doi.org/10.3758/BF03194066)


---

### Answer by username_2 on 2012-03-29. Score: 8

Bias can be quantified in many different ways. In human memory research carried out in the cognitive psychology tradition, there are simple ways to think about it. One basic measure of cognitive bias is merely called *bias*, and it's a measure of the absolute accuracy of an individual's probability judgments. You average probability judgments across a given subject, then average performance, and subtract one from the other. The magnitude and direction of this discrepancy could be argued to reflect a task-specific cognitive bias.

That's not useful without a concrete example. Say I ask you 10 questions tapping your general knowledge about the world (e.g., "Who invaded Rome by crossing the Alps with elephants?") After you respond, I ask you to rate your confidence on a scale from 0% (not at all confident) to 100% (entirely confident). We do this for nine more questions.

After you've taken this short test, I can easily calculate your bias by averaging your accuracy across the 10 questions (say you get 2 wrong, so 80%) and subtracting your average confidence across those questions (say it's 73%). It's a 7% difference, and you were more accurate than you were confident, so your bias is said to be 7% -- you were *underconfident*.

There are lots of other ways you can think about this. Say, for instance, you weren't interested in overconfidence or underconfidence, but rather, just the degree to which your performance differed from your estimations. Here you could just use the squared value of the difference to represent probability accuracy.

> **Comments:**
> **username_0**: Thanx, Andy. Just what I was asking for. Is there a general standard, or something almost everybody in the field uses?  
> **username_2**: Unfortunately I'm afraid it's really going to vary from discipline to discipline. For basic research, this was the most atheoretical topic I could come up with. I knew bias and similar measures are used whenever numerically-based probability judgments come into play (in the field of personality psychology, for instance). It's tougher for more qualitative areas, though.  

---

