<p>I'm not a psychometrician but I use these tools and have some familiarity with the topic so I thought I'd offer some brief thoughts.</p>
<p><strong>How does one validate a psychometric instrument?</strong></p>
<p>There are a few steps and this list isn't exhaustive:</p>
<ul>
<li>Conceptualise and clearly define a construct (not done particularly well in depression, which has changed definitions over time)</li>
<li>Develop questions/statements that can adequately identify the construct (see Eiko Fried's paper in my comment on the poor overlap between seven commonly used depression rating scales)</li>
<li>Establish norms and appropriate cut-offs (for depression, ensuring your criteria do not inappropriately diagnose normal sadness or neurotic/pessimistic dimensions of personality as depression)</li>
<li>Assess reliability between the individual items/questions (are the questions internally consistent?)</li>
<li>Assess construct validity (does your instrument correspond to similar instruments that have already been validated?)</li>
<li>Assess criterion validity (does your instrument correspond to the 'gold standard' for diagnosis?)</li>
<li>Assess test-retest reliability (does it measure the same thing over time?)</li>
<li>Use factor analysis to understand if there may be subtypes or dimensions to the construct</li>
</ul>
<p>Depression, as it's defined in the DSM-5-TR and the ICD-11 (the two most commonly used diagnostic manuals) has modest inter-rater reliability and fairly poor validity, in part because we don't have a robust theoretic model of depression, have no objective tests to identify the presence or absence of depression, and have significant methodological limitations affecting the instruments we use to measure depression (highlighted in your question). Currently, the 'gold standard' by which questionnaires are compared to is the opinion of a psychiatrist, which is subjective and prone to substantial noise.</p>
<p>I don't have good answers to many of the important questions you ask. I hope someone with specific psychometric experience will. I also hope that over time, with respect to depression symptom measurement, we'll move from discrete time point assessment of symptoms (e.g. you visit your doctor and see them again one month later) to ecological momentary assessment, involving multiple repeated measures of mood and behaviour from the same person in real-time to obtain a richer understanding of the emergence of trends and establishment of each individual's baseline.</p>

CC BY-SA 4.0
2024-01-12T06:57:32.453Z
2024-01-12T06:57:32.453Z
Malcolm Forbes
<p>Any science has to be built on the scientific method. While in all fields there are novel phenomena that are not yet understood, an essential step to doing robust science is understanding how one's instruments work, before using them to measure something more complicated.</p>
<p>For example, in physics, we might not yet know what is a Higgs boson, but we understand exactly how magnets and light detectors work. When we make an experiment and gather data, we can select the simplest model which explains all of the observed data, except for the error introduced by those magnets and light detectors, which can be precisely estimated.</p>
<p>Or, for example, in neuroscience, we still have very limited understanding how hippocampal ripples give rise to memory consolidation and what that even means, but the electrophysiological and optogenetic instruments are quite precise, we already sort of understand how single neurons work, and are continuing to iteratively improve instrumentation until we can record a sufficient amount of information from a brain area to fully be able to model what is happening inside of it.</p>
<p>I am trying to understand, where does psychometrics land in this regard. From what I have read, psychometricians use questionnaires as their instruments. I would like to know, how do psychometricians learn what exactly their instrument measure, and how do they estimate the accuracy of the instruments. Here are some naive questions from the top of my head</p>
<ol>
<li>Let's say I want to construct a question that measures whether a person is depressive. Sure, questions like &quot;are you sad often&quot; or &quot;do you find it hard to get out of bed in the morning&quot; sound plausible to me, but how do I test that they measure what I want? How do I even decide what exactly I want to measure? The definition of depression in DCM5 is a bit imprecise...</li>
<li>Let's say I know that my question measures depressiveness, but it also is influenced by a whole bunch of other stuff: Is the person honest; Are they generally an optimistic or pessimistic person; What mood are they in today; Do they understand the question; etc. How can I control for these confounds? How can I estimate the uncertainty in my instrument, in order to have an informed decision about the performance of a model I want to fit?</li>
<li>Let's say I have gathered a few questions that all measure depressiveness, and have some known uncertainty. I would proceed to fit some model in order to estimate depressiveness. I have heard that common models in modern psychometrics are Rasch models and their extensions. As far as I understand, these models assume that the uncertainty in the questions is caused by random noise, which is independent across questions. I am interested to know whether and how this assumption is typically justified. Naively, I would say that there is no randomness in this process - if asked the same question twice, most people would likely answer the same, if they have the minimal incentive to be honest and precise. Thus, any uncertainty is due to uncontrolled confounding factors. It is certainly plausible that a single confounding factor would affect multiple questions simultaneously. For example, a pessimist would likely write lower numbers for most questions even if they are perfectly healthy. Such confounding would result in correlated noise, and could significantly bias inference. How is this controlled for?</li>
</ol>

CC BY-SA 4.0
2024-01-12T06:08:35.397Z
These are excellent questions. I am not a psychometrician so am not best placed to answer but I recommend having a look at the work of Eiko Fried, starting with this paper - https://eiko-fried.com/wp-content/uploads/2022FriedDepressionMeasurement.pdf
Malcolm Forbes
CC BY-SA 4.0
2024-01-12T08:29:15.030Z
@MalcolmForbes Thanks :). Btw, I picked depression just as an example, it is not my ultimate goal. Still, a very interesting topic
Aleksejs Fomins
CC BY-SA 4.0
2024-01-12T09:24:12.247Z
@MalcolmForbes The publication by Eiko Fried is an interesting read. Indeed, they arrive at the main conclusion that having a solid mechanistic theory of the construct (in this case, biochemical/medical state of depression) is integral to understanding what exactly is being measured, and thus essential in constructing a valid and reliable metric.
Aleksejs Fomins
CC BY-SA 4.0
2024-01-12T19:15:39.990Z
I think the best way to learn *how people validate* these measures would be to read the papers where they are introduced. Then you can spend all day critiquing each of these. It's no easy task, and the measures used in research may not necessarily correspond well to clinical diagnosis.
Bryan Krause
CC BY-SA 4.0
2024-01-12T20:47:06.683Z
@BryanKrause, I am not looking for become an expert in the field. It does not matter to me how exactly the validation is done. To be more precise, I am interested to know what conditions need to be met for the validation to be possible at all, on a very abstract level. I have read a few papers on psychometrics so far, and in my experience, far more time is dedicated to expaining the methods in detail than trying to justify their use, so my time blindly reading them was not well spent
Aleksejs Fomins
CC BY-SA 4.0
2024-01-12T20:48:45.520Z
Well, your question asks how they do it; their methods section would be where you would read how they do it. It's common also in the Introduction or Discussion section of those papers to find criticism of other measures; there has to be some motivation for why they're working on a new one.
Bryan Krause
CC BY-SA 4.0
2024-01-12T20:51:25.037Z
https://web.archive.org/web/20170808035218id_/http://www.scalesandmeasures.net/files/files/Beck_AT_1961.pdf would be one paper for a very popular depression scale. https://web.archive.org/web/20170810064408id_/https://brainmass.com/file/238280/BDI_II+validity+and+reliability.pdf would be a more recent validation of the same scale in adolescents.
Bryan Krause
CC BY-SA 4.0
2024-01-12T20:52:37.870Z
@BryanKrause thanks, I will have a look. Let's see if their discussion sections are more detailed than what I have encountered so far
Aleksejs Fomins
CC BY-SA 4.0
2024-01-12T20:55:11.937Z
These particular examples may not be the best cases for finding arguments about other measures. But I do think the original BDI paper explains their validation procedure clearly.
Bryan Krause
<measurement><survey>
How does one validate a psychometric instrument
Aleksejs Fomins
# How does one validate a psychometric instrument

(Asked by: username_0 on 2024-01-11)

Any science has to be built on the scientific method. While in all fields there are novel phenomena that are not yet understood, an essential step to doing robust science is understanding how one's instruments work, before using them to measure something more complicated.

For example, in physics, we might not yet know what is a Higgs boson, but we understand exactly how magnets and light detectors work. When we make an experiment and gather data, we can select the simplest model which explains all of the observed data, except for the error introduced by those magnets and light detectors, which can be precisely estimated.

Or, for example, in neuroscience, we still have very limited understanding how hippocampal ripples give rise to memory consolidation and what that even means, but the electrophysiological and optogenetic instruments are quite precise, we already sort of understand how single neurons work, and are continuing to iteratively improve instrumentation until we can record a sufficient amount of information from a brain area to fully be able to model what is happening inside of it.

I am trying to understand, where does psychometrics land in this regard. From what I have read, psychometricians use questionnaires as their instruments. I would like to know, how do psychometricians learn what exactly their instrument measure, and how do they estimate the accuracy of the instruments. Here are some naive questions from the top of my head

1. Let's say I want to construct a question that measures whether a person is depressive. Sure, questions like "are you sad often" or "do you find it hard to get out of bed in the morning" sound plausible to me, but how do I test that they measure what I want? How do I even decide what exactly I want to measure? The definition of depression in DCM5 is a bit imprecise...
2. Let's say I know that my question measures depressiveness, but it also is influenced by a whole bunch of other stuff: Is the person honest; Are they generally an optimistic or pessimistic person; What mood are they in today; Do they understand the question; etc. How can I control for these confounds? How can I estimate the uncertainty in my instrument, in order to have an informed decision about the performance of a model I want to fit?
3. Let's say I have gathered a few questions that all measure depressiveness, and have some known uncertainty. I would proceed to fit some model in order to estimate depressiveness. I have heard that common models in modern psychometrics are Rasch models and their extensions. As far as I understand, these models assume that the uncertainty in the questions is caused by random noise, which is independent across questions. I am interested to know whether and how this assumption is typically justified. Naively, I would say that there is no randomness in this process - if asked the same question twice, most people would likely answer the same, if they have the minimal incentive to be honest and precise. Thus, any uncertainty is due to uncontrolled confounding factors. It is certainly plausible that a single confounding factor would affect multiple questions simultaneously. For example, a pessimist would likely write lower numbers for most questions even if they are perfectly healthy. Such confounding would result in correlated noise, and could significantly bias inference. How is this controlled for?

**Tags:** measurement, survey

> **Comments:**
> **username_1**: These are excellent questions. I am not a psychometrician so am not best placed to answer but I recommend having a look at the work of Eiko Fried, starting with this paper - https://eiko-fried.com/wp-content/uploads/2022FriedDepressionMeasurement.pdf  
> **username_0**: @MalcolmForbes Thanks :). Btw, I picked depression just as an example, it is not my ultimate goal. Still, a very interesting topic  
> **username_0**: @MalcolmForbes The publication by Eiko Fried is an interesting read. Indeed, they arrive at the main conclusion that having a solid mechanistic theory of the construct (in this case, biochemical/medical state of depression) is integral to understanding what exactly is being measured, and thus essential in constructing a valid and reliable metric.  
> **username_2**: I think the best way to learn *how people validate* these measures would be to read the papers where they are introduced. Then you can spend all day critiquing each of these. It's no easy task, and the measures used in research may not necessarily correspond well to clinical diagnosis.  
> **username_0**: @BryanKrause, I am not looking for become an expert in the field. It does not matter to me how exactly the validation is done. To be more precise, I am interested to know what conditions need to be met for the validation to be possible at all, on a very abstract level. I have read a few papers on psychometrics so far, and in my experience, far more time is dedicated to expaining the methods in detail than trying to justify their use, so my time blindly reading them was not well spent  
> **username_2**: Well, your question asks how they do it; their methods section would be where you would read how they do it. It's common also in the Introduction or Discussion section of those papers to find criticism of other measures; there has to be some motivation for why they're working on a new one.  
> **username_2**: https://web.archive.org/web/20170808035218id_/http://www.scalesandmeasures.net/files/files/Beck_AT_1961.pdf would be one paper for a very popular depression scale. https://web.archive.org/web/20170810064408id_/https://brainmass.com/file/238280/BDI_II+validity+and+reliability.pdf would be a more recent validation of the same scale in adolescents.  
> **username_0**: @BryanKrause thanks, I will have a look. Let's see if their discussion sections are more detailed than what I have encountered so far  
> **username_2**: These particular examples may not be the best cases for finding arguments about other measures. But I do think the original BDI paper explains their validation procedure clearly.  

---

## 1 answers

### Answer by username_1 on 2024-01-12. Score: 1

I'm not a psychometrician but I use these tools and have some familiarity with the topic so I thought I'd offer some brief thoughts.

**How does one validate a psychometric instrument?**

There are a few steps and this list isn't exhaustive:

* Conceptualise and clearly define a construct (not done particularly well in depression, which has changed definitions over time)
* Develop questions/statements that can adequately identify the construct (see Eiko Fried's paper in my comment on the poor overlap between seven commonly used depression rating scales)
* Establish norms and appropriate cut-offs (for depression, ensuring your criteria do not inappropriately diagnose normal sadness or neurotic/pessimistic dimensions of personality as depression)
* Assess reliability between the individual items/questions (are the questions internally consistent?)
* Assess construct validity (does your instrument correspond to similar instruments that have already been validated?)
* Assess criterion validity (does your instrument correspond to the 'gold standard' for diagnosis?)
* Assess test-retest reliability (does it measure the same thing over time?)
* Use factor analysis to understand if there may be subtypes or dimensions to the construct

Depression, as it's defined in the DSM-5-TR and the ICD-11 (the two most commonly used diagnostic manuals) has modest inter-rater reliability and fairly poor validity, in part because we don't have a robust theoretic model of depression, have no objective tests to identify the presence or absence of depression, and have significant methodological limitations affecting the instruments we use to measure depression (highlighted in your question). Currently, the 'gold standard' by which questionnaires are compared to is the opinion of a psychiatrist, which is subjective and prone to substantial noise.

I don't have good answers to many of the important questions you ask. I hope someone with specific psychometric experience will. I also hope that over time, with respect to depression symptom measurement, we'll move from discrete time point assessment of symptoms (e.g. you visit your doctor and see them again one month later) to ecological momentary assessment, involving multiple repeated measures of mood and behaviour from the same person in real-time to obtain a richer understanding of the emergence of trends and establishment of each individual's baseline.


---

