<p>I have two versions of calculating a score for a test (which assesses how well they perform a task). One takes into account the difficulty of the items, the other does not. These tests are administered at different times. Each test is independent from the other. I expect some subjects to learn the task fairly quickly and others might not. So this test isn't really measuring a trait and I assume metrics like test-retest reliability don't really apply here.</p>
<p>How would I go about demonstrating that my test which takes into account the difficulty of the items is a more &quot;fair&quot; representation of performance than one which does not?</p>

CC BY-SA 4.0
2022-05-09T04:31:23.307Z
You have only one measure of each for each subject? Do you measure anything else useful, like a real-world outcome?
Bryan Krause
CC BY-SA 4.0
2022-05-09T05:18:51.740Z
A subject can be tested multiple times. Each time the subset of items will be different. There will be little or no overlap between the items across subjects.
user2855666
CC BY-SA 4.0
2022-05-10T08:52:43.077Z
Welcome. The question is unclear. In the title you inquire about test validity and reliability. In the last line you want to show test B is better than A.  *Validity* means to show a test outcome is what you want to measure, this requires a gold standard test outcome where your two tests are benchmarked / compared against. *Reliability* is basically whether the test is consistent across different occasions of testing, this necessitates repeated measurements of each two individual tests. 'Proving' whether *one test is better than the other* can be dependent on a great many things....
AliceD
CC BY-SA 4.0
2022-05-10T08:55:06.177Z
...So basically I see three questions mixed together in a crock pot. By the looks of it, you need to first figure out what the research question was that you wish to answer with this experiment. After that, think about how you want to answer it and if you have gathered enough data. As of now, this answer is pretty much unanswerable imo.
AliceD
CC BY-SA 4.0
2022-05-10T15:45:32.160Z
The question is about reliability. It seems as though when test-retest reliability is measured the same test is administered to multiple subjects on multiple occasions. In this case, each subject is presented with a different test (both at test and retest). However, we have a proxy for the difficulty of the items in the tests that were administered. How would you go about assessing test-retest reliability in this situation?
user2855666
CC BY-SA 4.0
2022-05-10T15:48:07.030Z
Overall, the question isn't so much about which test is better than the other but rather whether the more "fair" version of the test is more reliable. There's a way to assess validity here. Let's focus on the reliability part. The key is that the items are not the same across subjects. Also, we're not really measuring a trait here so maybe we don't expect subject performance to be stable across time?
user2855666
CC BY-SA 4.0
2022-05-10T16:14:53.180Z
Test-retest reliability is just that; if you want to compare your two tests, you'd repeat each multiple times and see if Test A is more reliable within Test A than Test B is reliable within Test B. I don't see anything special here that makes it different that one takes difficulty into account and the other doesn't, you'd treat them just as you would any other two tests.
Bryan Krause
CC BY-SA 4.0
2022-05-10T16:15:36.187Z
Also, if your question is about reliability, why do you mention validity? And if you expect scores to change (for example, due to learning), you may not actually want to measure test-retest reliability.
Bryan Krause
CC BY-SA 4.0
2022-05-10T16:36:54.480Z
What is special here is that for measures of test-retest reliability the test you provide to all subjects is the same for each presentation of that test. Here the items are different across subjects. What are the implications? Are there alternative measures than just calculating the correlation across subjects? (between test and retest)
I mention validity because my question is about all possible metrics to assess whether one version of the test (which accounts for difficulty) is better or more "fair" than the other. This would also include validity.
user2855666
CC BY-SA 4.0
2022-05-10T16:38:47.953Z
To address your second point. I assume when people look at test-retest reliability they factor in learning somehow. Or if they don't even look at test-retest reliability where there's learning then how do they assess the reliability of a test when there is some learning?
user2855666
CC BY-SA 4.0
2022-05-11T04:36:25.760Z
I don't agree with "for measures of test-retest reliability the test you provide to all subjects is the same for each presentation of that test"; this is not necessary. "I assume when people look at test-retest reliability they factor in learning somehow." Not typically, though you may consider comparison within a group that's had the same opportunity to learn, and consider a test reliable if the adjusted scores are consistent.
Bryan Krause
CC BY-SA 4.0
2022-05-18T18:24:26.310Z
It's not about whether we believe it's necessary or not. Of course you can calculate test-retest reliability on any two samples, and that's just saying that you can calculate correlations between any two samples. But I don't know of any example of this and I suspect it's for a very good reason, namely: if each individual test isn't the same across subjects (at least in terms of the difficulty of the items) then this correlation will be misleading/meaningless.
user2855666
<learning><statistics><psychology><test><educational-psychology>
Alternative metrics to assess test validity/reliability
user2855666
# Alternative metrics to assess test validity/reliability

(Asked by: username_0 on 2022-05-09)

I have two versions of calculating a score for a test (which assesses how well they perform a task). One takes into account the difficulty of the items, the other does not. These tests are administered at different times. Each test is independent from the other. I expect some subjects to learn the task fairly quickly and others might not. So this test isn't really measuring a trait and I assume metrics like test-retest reliability don't really apply here.

How would I go about demonstrating that my test which takes into account the difficulty of the items is a more "fair" representation of performance than one which does not?

**Tags:** learning, statistics, psychology, test, educational-psychology

> **Comments:**
> **username_1**: You have only one measure of each for each subject? Do you measure anything else useful, like a real-world outcome?  
> **username_0**: A subject can be tested multiple times. Each time the subset of items will be different. There will be little or no overlap between the items across subjects.  
> **username_2**: Welcome. The question is unclear. In the title you inquire about test validity and reliability. In the last line you want to show test B is better than A.  *Validity* means to show a test outcome is what you want to measure, this requires a gold standard test outcome where your two tests are benchmarked / compared against. *Reliability* is basically whether the test is consistent across different occasions of testing, this necessitates repeated measurements of each two individual tests. 'Proving' whether *one test is better than the other* can be dependent on a great many things....  
> **username_2**: ...So basically I see three questions mixed together in a crock pot. By the looks of it, you need to first figure out what the research question was that you wish to answer with this experiment. After that, think about how you want to answer it and if you have gathered enough data. As of now, this answer is pretty much unanswerable imo.  
> **username_0**: The question is about reliability. It seems as though when test-retest reliability is measured the same test is administered to multiple subjects on multiple occasions. In this case, each subject is presented with a different test (both at test and retest). However, we have a proxy for the difficulty of the items in the tests that were administered. How would you go about assessing test-retest reliability in this situation?  
> **username_0**: Overall, the question isn't so much about which test is better than the other but rather whether the more "fair" version of the test is more reliable. There's a way to assess validity here. Let's focus on the reliability part. The key is that the items are not the same across subjects. Also, we're not really measuring a trait here so maybe we don't expect subject performance to be stable across time?  
> **username_1**: Test-retest reliability is just that; if you want to compare your two tests, you'd repeat each multiple times and see if Test A is more reliable within Test A than Test B is reliable within Test B. I don't see anything special here that makes it different that one takes difficulty into account and the other doesn't, you'd treat them just as you would any other two tests.  
> **username_1**: Also, if your question is about reliability, why do you mention validity? And if you expect scores to change (for example, due to learning), you may not actually want to measure test-retest reliability.  
> **username_0**: What is special here is that for measures of test-retest reliability the test you provide to all subjects is the same for each presentation of that test. Here the items are different across subjects. What are the implications? Are there alternative measures than just calculating the correlation across subjects? (between test and retest) I mention validity because my question is about all possible metrics to assess whether one version of the test (which accounts for difficulty) is better or more "fair" than the other. This would also include validity.  
> **username_0**: To address your second point. I assume when people look at test-retest reliability they factor in learning somehow. Or if they don't even look at test-retest reliability where there's learning then how do they assess the reliability of a test when there is some learning?  
> **username_1**: I don't agree with "for measures of test-retest reliability the test you provide to all subjects is the same for each presentation of that test"; this is not necessary. "I assume when people look at test-retest reliability they factor in learning somehow." Not typically, though you may consider comparison within a group that's had the same opportunity to learn, and consider a test reliable if the adjusted scores are consistent.  
> **username_0**: It's not about whether we believe it's necessary or not. Of course you can calculate test-retest reliability on any two samples, and that's just saying that you can calculate correlations between any two samples. But I don't know of any example of this and I suspect it's for a very good reason, namely: if each individual test isn't the same across subjects (at least in terms of the difficulty of the items) then this correlation will be misleading/meaningless.  

---

## 0 answers

