<p>Harvill mentions an estimate by Lord (1959). Lord (1959) presents some data for the standard error of measurement for some moderately difficult cognitive measures. While there are many caveats (e.g., the estimate of the standard error is most accurate for scores around 50% and the estimates are based on tests that are neither particularly easy or particularly difficult with means in the .35 to .75 range), Lord provides a simple formula that can be used as a rule of thumb for predicting standard error of measurement in his sample of cognitive measures which performed quite well. </p>

<p>$$\hat{s}_e = .432 \sqrt{k}$$</p>

<p>where $k$ is the number of items.
Alternatively, if you are interested in the mean correct on a 0 to 100 scale rather than the total correct, you can divide by $k$ and multiply by 100.</p>

<p>$$\hat{s}_e = \frac{.432 \sqrt{k}}{k} \times 100$$</p>

<p>When I plugged this into R for some sample values I obtained:</p>

<pre><code>&gt; lord_approximation &lt;- function(k) 0.432 * sqrt(k) /k * 100
&gt; k &lt;- c(10, 20, 50, 80, 100, 120, 150, 200)
&gt; cbind(k, sem=round(lord_approximation(k), 2))
       k   sem
[1,]  10 13.66
[2,]  20  9.66
[3,]  50  6.11
[4,]  80  4.83
[5,] 100  4.32
[6,] 120  3.94
[7,] 150  3.53
[8,] 200  3.05
</code></pre>

<p>Of course, not all of this reduction in standard error of measurement is due to greater accuracy. Some of it comes from the smaller standard deviation in true scores that occurs when you take the mean of more items. Furthermore, these estimates are based on relatively well designed cognitive measures. Teacher designed tests may have slightly lower reliability and thus larger SEM.</p>

<h3>References</h3>

<ul>
<li>Harvill, L. M. (1991). Standard error of measurement. Educational Measurement: Issues and Practice, 10(2), 33-41. <a href="http://ncme.org/linkservid/6606715E-1320-5CAE-6E9DDC581EE47F88/showMeta/0/" rel="nofollow">PDF</a></li>
<li>Lord, F. M. (1959). Tests of the same length do have the same standard 
error of measurement. Educational and Psychological Measurement, 19, 233-239.</li>
</ul>

CC BY-SA 3.0
2013-03-14T12:52:43.860Z
interesting, but this seems to assume that each question is independent of the others. However, in an actual test there is extremely high correlations between questions. In particular, a valid measure should at least account for questions coming from some fixed number of relatively-independent units (i.e. chapters or topics) with high question-question correlation within the units. You could also give teachers a software tool to infer typical question-question correlations by doing statistics on student results on their previous exams.
Artem Kaznatcheev
CC BY-SA 3.0
2013-03-08T06:27:52.213Z
2013-03-08T06:27:52.213Z
Jeromy Anglim
<p>To me the most natural solution is to just use item response theory (IRT).  IRT has been around for a few decades, so it is well established, implemented in a variety of software packages and provides a sensible, extensible framework for this type of problem.</p>

<p>Essentially, one assumes an underlying latent construct of interest, values of which should drive responses on the test.  For multiple choice where the answer is "right" or "wrong", you can use a series of logit (canonically) or probit models.  Then for each student, you can estimate the score on the underlying latent variable and that will naturally come with some estimate of its quality/variability.</p>

<p>Issues that are automatically handled:</p>

<ul>
<li>If everyone (or nearly everyone) gets an answer correct, it contributes very little information</li>
<li>Corollary to #1, if almost no one gets an answer correct and someone does, it should be weighted more heavily.  Essentially, item difficulty is automatically handled.</li>
<li>Inter-dependence among items is accounted for.  Asking the same item 10 times in a row will not artificially decrease your measurement error.</li>
</ul>

<p>If you take a Bayesian view, for each student, you could use the model and their test responses to calculate a posterior distribution for the latent construct of interest, which would allow both a point estimate (e.g., posterior mean, median, or mode), as well as estimates of variability (e.g., standard deviation; 95% high posterior density region).</p>

<p>This sort of thing is essentially what big nation wide tests and testing services do.  It is actually not too hard to do, but probably enough effort most teachers who already feel overworked do not adopt them.</p>

CC BY-SA 3.0
2013-03-15T18:50:31.600Z
Solid answer, maybe teachers would use this if you made a plug-and-play version of this that directly takes the results from scantron sheets (are those still in use?) and does the statistics for them?
Artem Kaznatcheev
CC BY-SA 3.0
2013-03-17T23:55:03.707Z
+1 I agree IRT (bayesian IRT in particular) provides a great framework for modelling multiple choice tests. In my particular case, I'm looking for heuristics that teachers might use to guide their decision about how many items to include in a test. I'd like to be able to give teachers a rough sense of how much more accurate their test could be if they for example increased their test from 80 to 100 items.
Jeromy Anglim
CC BY-SA 3.0
2013-03-18T00:54:50.200Z
In addition to difficulties in implementation, I think IRT is not applied in teacher constructed test settings because there is often an implicit contract with the students. That is, a student's mark is directly related to the proportion of items answered correctly. Thus, in IRT there is still the question of how thetas will be mapped on to exam grades. Of course this can be overcome, but I think it presents another obstacle to uptake, particularly where the test is used to make judgements about absolute performance and not just normative performance.
Jeromy Anglim
CC BY-SA 3.0
2013-03-24T04:29:45.007Z
@JeromyAnglim I agree with that, with respect to absolute performance.  However, the implicit assumption there is that teachers actually have a valid measure of absolute performance.  I think that is a _much_ stricter assumption than that they have a valid measure of _relative_ performance (I am open to being argued out of that).  Regarding the accuracy of the test, I am not sure how reasonable any heuristic could be. That is dependent on the relative (in)dependence of items, right? In the extreme, can a teacher ask the same question another 20 times to get to 100 and still be more accurate?
Joshua
CC BY-SA 3.0
2013-03-24T12:05:18.610Z
With regards to heuristics: Any heuristic would be based on the assumption that the teacher would be writing a set of additional items of roughly comparable quality, difficulty, and variety as the existing items. I acknowledge that some teachers might differ in their skill in doing this, but at the same time, a heuristic could be frame in terms of a ball park or a typical range.
Jeromy Anglim
CC BY-SA 3.0
2013-03-24T12:08:03.183Z
With regards to absolute performance: I agree that making claims about absolute performance from a test in a rigorous way is very difficult. But ultimately any teacher who is asked to decide what is the passing grade is forced to draw an absolute line in the sand somewhere. This might be a particular value of theta or a proportion correct on a test. My guess is that many teachers feel more comfortable talking about proportion correct (e.g., less than 50%, 60%, or whatever is the custom is a fail).
Jeromy Anglim
CC BY-SA 3.0
2013-03-15T18:36:02.093Z
2013-03-15T18:36:02.093Z
Joshua
<p>Assume a teacher constructs a four-choice multiple choice test. Each item has only one correct response. The test is scored from 0 to 100 representing the percentage of items answered correctly. </p>

<p>I want to have some rules of thumb that could be informative regarding how many items are required to achieve a given standard error of measurement. For example, it would be nice to be able to advice teachers who write their own multiple choice exams "if you have 100 items that are reasonably well worded, you can expect a standard error of measurement of 2.5". </p>

<p>The standard error of measurement is often defined as:</p>

<p>$$s_e=s_x \sqrt{1-r_{xx}}$$</p>

<p>where $s_x$ is the standard deviation and $r_{xx}$ is the reliability.</p>

<p>Furthermore internal consistency reliability can be calculated from the number of items $k$ and the mean inter-item correlation $\bar{r}_{ij}$ (i.e., average correlation between item $i$ and item $j$ for all $k$ items where $i\neq j$):</p>

<p>$$r_{xx}=\frac{k(\bar{r}_{ij})}
{1 + (k -1) \bar{r}_{ij}}$$</p>

<p>However, I'm wanting to translate the above information into meaningful recommendations for teachers. Thus, this assumes that I have some empirical estimate of typical values of $\bar{r}_{ij}$ and that I have an estimate of $s_x$. It then requires application of the formulas to calculate standard errors of measurement for likely numbers of items $k$. In particular, I was thinking about numbers of items equal to: 10, 20, 50, 80, 100, 120, 150, and 200.</p>

<p>Thus, I was wondering whether there are any published estimates of the standard error of measurement teacher constructed multiple choice tests.</p>

CC BY-SA 3.0
2013-03-10T23:31:10.910Z
Standard error of measurement is the standard deviation that would be obtained if you were able to repeatedly obtain a measure for a particular individual under hypothetical identical circumstances. I.e., it's a measure of the uncertainty you have about a measure you have obtained on a person.
Jeromy Anglim
CC BY-SA 3.0
2013-03-14T12:48:12.430Z
This is awesome! It'd be great if I received all my exams and assignments back with error bars.
Artem Kaznatcheev
CC BY-SA 3.0
2013-03-15T18:39:18.983Z
A major issue here is how related are the items?  In particular, I would worry that the item similarity increases as the number increases --- a 20 item test may be rather different, but a 200 item test may have substantial redundancy.
Joshua
<measurement><test><educational-psychology><reliability>
What is the standard error of measurement for teacher made multiple choice tests?
Jeromy Anglim
# What is the standard error of measurement for teacher made multiple choice tests?

(Asked by: username_0 on 2013-03-08)

Assume a teacher constructs a four-choice multiple choice test. Each item has only one correct response. The test is scored from 0 to 100 representing the percentage of items answered correctly.

I want to have some rules of thumb that could be informative regarding how many items are required to achieve a given standard error of measurement. For example, it would be nice to be able to advice teachers who write their own multiple choice exams "if you have 100 items that are reasonably well worded, you can expect a standard error of measurement of 2.5".

The standard error of measurement is often defined as:

$$s\_e=s\_x \sqrt{1-r\_{xx}}$$

where $s\_x$ is the standard deviation and $r\_{xx}$ is the reliability.

Furthermore internal consistency reliability can be calculated from the number of items $k$ and the mean inter-item correlation $\bar{r}\_{ij}$ (i.e., average correlation between item $i$ and item $j$ for all $k$ items where $i\neq j$):

$$r\_{xx}=\frac{k(\bar{r}\_{ij})}
{1 + (k -1) \bar{r}\_{ij}}$$

However, I'm wanting to translate the above information into meaningful recommendations for teachers. Thus, this assumes that I have some empirical estimate of typical values of $\bar{r}\_{ij}$ and that I have an estimate of $s\_x$. It then requires application of the formulas to calculate standard errors of measurement for likely numbers of items $k$. In particular, I was thinking about numbers of items equal to: 10, 20, 50, 80, 100, 120, 150, and 200.

Thus, I was wondering whether there are any published estimates of the standard error of measurement teacher constructed multiple choice tests.

**Tags:** measurement, test, educational-psychology, reliability

> **Comments:**
> **username_0**: Standard error of measurement is the standard deviation that would be obtained if you were able to repeatedly obtain a measure for a particular individual under hypothetical identical circumstances. I.e., it's a measure of the uncertainty you have about a measure you have obtained on a person.  
> **username_1**: This is awesome! It'd be great if I received all my exams and assignments back with error bars.  
> **username_2**: A major issue here is how related are the items?  In particular, I would worry that the item similarity increases as the number increases --- a 20 item test may be rather different, but a 200 item test may have substantial redundancy.  

---

## 2 answers

### Answer by username_0 on 2013-03-08. Score: 3

Harvill mentions an estimate by Lord (1959). Lord (1959) presents some data for the standard error of measurement for some moderately difficult cognitive measures. While there are many caveats (e.g., the estimate of the standard error is most accurate for scores around 50% and the estimates are based on tests that are neither particularly easy or particularly difficult with means in the .35 to .75 range), Lord provides a simple formula that can be used as a rule of thumb for predicting standard error of measurement in his sample of cognitive measures which performed quite well. 
$$\hat{s}_e = .432 \sqrt{k}$$
where $k$ is the number of items.
Alternatively, if you are interested in the mean correct on a 0 to 100 scale rather than the total correct, you can divide by $k$ and multiply by 100.
$$\hat{s}_e = \frac{.432 \sqrt{k}}{k} \times 100$$
When I plugged this into R for some sample values I obtained:
> lord_approximation <- function(k) 0.432 * sqrt(k) /k * 100
> k <- c(10, 20, 50, 80, 100, 120, 150, 200)
> cbind(k, sem=round(lord_approximation(k), 2))
       k   sem
[1,]  10 13.66
[2,]  20  9.66
[3,]  50  6.11
[4,]  80  4.83
[5,] 100  4.32
[6,] 120  3.94
[7,] 150  3.53
[8,] 200  3.05

Of course, not all of this reduction in standard error of measurement is due to greater accuracy. Some of it comes from the smaller standard deviation in true scores that occurs when you take the mean of more items. Furthermore, these estimates are based on relatively well designed cognitive measures. Teacher designed tests may have slightly lower reliability and thus larger SEM.
References

Harvill, L. M. (1991). Standard error of measurement. Educational Measurement: Issues and Practice, 10(2), 33-41. PDF
Lord, F. M. (1959). Tests of the same length do have the same standard 
error of measurement. Educational and Psychological Measurement, 19, 233-239.

> **Comments:**
> **username_1**: interesting, but this seems to assume that each question is independent of the others. However, in an actual test there is extremely high correlations between questions. In particular, a valid measure should at least account for questions coming from some fixed number of relatively-independent units (i.e. chapters or topics) with high question-question correlation within the units. You could also give teachers a software tool to infer typical question-question correlations by doing statistics on student results on their previous exams.  

---

### Answer by username_2 on 2013-03-15. Score: 2

To me the most natural solution is to just use item response theory (IRT). IRT has been around for a few decades, so it is well established, implemented in a variety of software packages and provides a sensible, extensible framework for this type of problem.

Essentially, one assumes an underlying latent construct of interest, values of which should drive responses on the test. For multiple choice where the answer is "right" or "wrong", you can use a series of logit (canonically) or probit models. Then for each student, you can estimate the score on the underlying latent variable and that will naturally come with some estimate of its quality/variability.

Issues that are automatically handled:

* If everyone (or nearly everyone) gets an answer correct, it contributes very little information
* Corollary to #1, if almost no one gets an answer correct and someone does, it should be weighted more heavily. Essentially, item difficulty is automatically handled.
* Inter-dependence among items is accounted for. Asking the same item 10 times in a row will not artificially decrease your measurement error.

If you take a Bayesian view, for each student, you could use the model and their test responses to calculate a posterior distribution for the latent construct of interest, which would allow both a point estimate (e.g., posterior mean, median, or mode), as well as estimates of variability (e.g., standard deviation; 95% high posterior density region).

This sort of thing is essentially what big nation wide tests and testing services do. It is actually not too hard to do, but probably enough effort most teachers who already feel overworked do not adopt them.

> **Comments:**
> **username_1**: Solid answer, maybe teachers would use this if you made a plug-and-play version of this that directly takes the results from scantron sheets (are those still in use?) and does the statistics for them?  
> **username_0**: +1 I agree IRT (bayesian IRT in particular) provides a great framework for modelling multiple choice tests. In my particular case, I'm looking for heuristics that teachers might use to guide their decision about how many items to include in a test. I'd like to be able to give teachers a rough sense of how much more accurate their test could be if they for example increased their test from 80 to 100 items.  
> **username_0**: In addition to difficulties in implementation, I think IRT is not applied in teacher constructed test settings because there is often an implicit contract with the students. That is, a student's mark is directly related to the proportion of items answered correctly. Thus, in IRT there is still the question of how thetas will be mapped on to exam grades. Of course this can be overcome, but I think it presents another obstacle to uptake, particularly where the test is used to make judgements about absolute performance and not just normative performance.  
> **username_2**: @JeromyAnglim I agree with that, with respect to absolute performance.  However, the implicit assumption there is that teachers actually have a valid measure of absolute performance.  I think that is a _much_ stricter assumption than that they have a valid measure of _relative_ performance (I am open to being argued out of that).  Regarding the accuracy of the test, I am not sure how reasonable any heuristic could be. That is dependent on the relative (in)dependence of items, right? In the extreme, can a teacher ask the same question another 20 times to get to 100 and still be more accurate?  
> **username_0**: With regards to heuristics: Any heuristic would be based on the assumption that the teacher would be writing a set of additional items of roughly comparable quality, difficulty, and variety as the existing items. I acknowledge that some teachers might differ in their skill in doing this, but at the same time, a heuristic could be frame in terms of a ball park or a typical range.  
> **username_0**: With regards to absolute performance: I agree that making claims about absolute performance from a test in a rigorous way is very difficult. But ultimately any teacher who is asked to decide what is the passing grade is forced to draw an absolute line in the sand somewhere. This might be a particular value of theta or a proportion correct on a test. My guess is that many teachers feel more comfortable talking about proportion correct (e.g., less than 50%, 60%, or whatever is the custom is a fail).  

---

