<p>Tough question.</p>

<p>I'm going to assume that the question pertains to measures which lack all the information you've outlined: test manual, validity studies, website and key articles, item content and normative comparisons. If these assumptions are correct, then what you are a describing is an insufficiently explained measure. I will argue that we necessarily cannot know what such a scale means in any scientific sense.</p>

<p>A scale is a measure. A <em>scientific</em> measure is a process which generates evidence, i.e., observations that discriminate between different possible answers to a specific question. (Data may constitute evidence with respect to one question and not another.) The outcome of this discrimination, or analysis, is what the scale "means", i.e., the logically sound interpretation we may make on the basis of analyzing the evidence.</p>

<p>To determine whether a method does, in fact, generate evidence, we need to be able to replicate it independently. But the premises imply that we don't have enough information to do so. Because we cannot independently verify which hypotheses the evidence generated by the method discriminates between, if any, there is no way to know what it means. </p>

CC BY-SA 3.0
2015-05-09T17:24:11.760Z
First time I did a pure methodology answer, so call it a test. I don't feel too good about the lack of references for the evidence definition, but I don't even know where I'd start tracking them down. Thoughts, comments, downvotes welcome.
Christian Hummeluhr
CC BY-SA 3.0
2015-05-09T20:43:04.893Z
I tend to agree with the conclusion that a normative or absolute understanding of the test is impossible given the premises that are in the question. I think the best solution -- it's not a very good one -- is to collect the data yourself.
Josh
CC BY-SA 3.0
2015-05-10T07:05:58.317Z
Thanks for coming back to this question. To clarify, I didn't mean to imply in the question that item information was not available anywhere. I guess the question is referring to the general situation where there probably are previous uses of the measure in the literature, and there might be a key article that describes the measure somewhere. I guess I'm thinking of a good general strategy for securing relevant information and then using it to form an understanding of the scale properties.
Jeromy Anglim
CC BY-SA 3.0
2015-05-10T07:33:47.263Z
@JeromyAnglim So, just to clarify ... something like (1) What information do we need to interpret a scale? (2) How can we most effectively get that information?
Christian Hummeluhr
CC BY-SA 3.0
2015-05-10T07:45:57.673Z
@ChristianHummeluhr Yep :-)
Jeromy Anglim
CC BY-SA 3.0
2015-05-09T17:06:25.077Z
2015-05-09T17:14:40.183Z
2015-05-09T17:14:40.183Z
Christian Hummeluhr
<p><strong>Context:</strong> When reading research articles in psychology, you often encounter new non-aptitude self-report psychological tests (e.g., measures of personality, well-being, psychopathology, learning style, stress, goal orientation, etc.). Most of these tests involve obtaining the sum or the mean of a set of items for one or more scales or subscales. Each item typically involves item text and a consistent set of response options (e.g., Likert-type response scales, yes-no response scales, etc.). There are thousands of such scales, some which are frequently used; others are used only once.</p>

<p>A lot has been written about how to assess <a href="http://en.wikipedia.org/wiki/Test_validity" rel="nofollow">test validity</a>. However, here I wanted to focus particularly on how to assess what a particular individual or group score means. Specifically, I'm thinking of the situation where you are reading a journal article and you encounter a scale that you have not seen before, such as:</p>

<ul>
<li>A 10-item measure of self-esteem on 1 to 5 scale; group mean is 3.2. Is the sample generally high, low, or moderate in self-esteem?</li>
<li>A 30-item check-list of good eating habits with a mean of 20.7. Does the sample have good eating habits? </li>
</ul>

<p>For well developed tests, there is a test manual. Other times there is a website or key journal article with information. Finally, in some cases, very little information is readily available. In some cases useful information is given in the methods section, but typically, on its own, this is inadequate. 
Thus, the process of trying to locate information such as item content and normative comparisons begin.  </p>

<p>A lot has been said about <a href="http://en.wikipedia.org/wiki/Norm-referenced_test" rel="nofollow">normative</a> and <a href="http://en.wikipedia.org/wiki/Criterion-referenced_assessment" rel="nofollow">criterion referenced test</a> interpretation.  If you are using a test in clinical practice or you are using a test in your own research, you should invest time in understanding what different scores on the test actually mean.   However, when casually reading a journal article you can encounter many tests. 
In such cases, the article is one of many, and you might just want to get a quick sense of what a given score means.</p>

<h3>Question</h3>

<ul>
<li><strong>What is a good way to get both a normative and absolute understanding of the meaning of a group or individual non-aptitude test score?</strong></li>
<li><strong>What is an efficient way of obtaining the information required  to make such a judgement?</strong></li>
</ul>

<p>Note the use of the term "non-aptitude". I'd like to exclude discussion of intelligence testing, ability testing, and knowledge testing, and so on.</p>

<p><strong>Update:</strong> To clarify, I didn't mean to imply in the question that item information was not available anywhere. I guess the question is referring to the general situation where there probably are previous uses of the measure in the literature, and there might be a key article that describes the measure somewhere. I guess I'm thinking of a good general strategy for securing relevant information and then using it to form an understanding of the scale properties.</p>

<measurement><methodology><test>
How to efficiently get a sense of the meaning of a score on a non-aptitude test that you encounter in a journal article?
Jeromy Anglim
# How to efficiently get a sense of the meaning of a score on a non-aptitude test that you encounter in a journal article?

(Asked by: username_0 on 2012-07-07)

**Context:** When reading research articles in psychology, you often encounter new non-aptitude self-report psychological tests (e.g., measures of personality, well-being, psychopathology, learning style, stress, goal orientation, etc.). Most of these tests involve obtaining the sum or the mean of a set of items for one or more scales or subscales. Each item typically involves item text and a consistent set of response options (e.g., Likert-type response scales, yes-no response scales, etc.). There are thousands of such scales, some which are frequently used; others are used only once.

A lot has been written about how to assess [test validity](http://en.wikipedia.org/wiki/Test_validity). However, here I wanted to focus particularly on how to assess what a particular individual or group score means. Specifically, I'm thinking of the situation where you are reading a journal article and you encounter a scale that you have not seen before, such as:

* A 10-item measure of self-esteem on 1 to 5 scale; group mean is 3.2. Is the sample generally high, low, or moderate in self-esteem?
* A 30-item check-list of good eating habits with a mean of 20.7. Does the sample have good eating habits?

For well developed tests, there is a test manual. Other times there is a website or key journal article with information. Finally, in some cases, very little information is readily available. In some cases useful information is given in the methods section, but typically, on its own, this is inadequate.
Thus, the process of trying to locate information such as item content and normative comparisons begin.

A lot has been said about [normative](http://en.wikipedia.org/wiki/Norm-referenced_test) and [criterion referenced test](http://en.wikipedia.org/wiki/Criterion-referenced_assessment) interpretation. If you are using a test in clinical practice or you are using a test in your own research, you should invest time in understanding what different scores on the test actually mean. However, when casually reading a journal article you can encounter many tests.
In such cases, the article is one of many, and you might just want to get a quick sense of what a given score means.

### Question

* **What is a good way to get both a normative and absolute understanding of the meaning of a group or individual non-aptitude test score?**
* **What is an efficient way of obtaining the information required to make such a judgement?**

Note the use of the term "non-aptitude". I'd like to exclude discussion of intelligence testing, ability testing, and knowledge testing, and so on.

**Update:** To clarify, I didn't mean to imply in the question that item information was not available anywhere. I guess the question is referring to the general situation where there probably are previous uses of the measure in the literature, and there might be a key article that describes the measure somewhere. I guess I'm thinking of a good general strategy for securing relevant information and then using it to form an understanding of the scale properties.

**Tags:** measurement, methodology, test


---

## 1 answers

### Answer by username_1 on 2015-05-09. Score: 2

Tough question.

I'm going to assume that the question pertains to measures which lack all the information you've outlined: test manual, validity studies, website and key articles, item content and normative comparisons. If these assumptions are correct, then what you are a describing is an insufficiently explained measure. I will argue that we necessarily cannot know what such a scale means in any scientific sense.

A scale is a measure. A *scientific* measure is a process which generates evidence, i.e., observations that discriminate between different possible answers to a specific question. (Data may constitute evidence with respect to one question and not another.) The outcome of this discrimination, or analysis, is what the scale "means", i.e., the logically sound interpretation we may make on the basis of analyzing the evidence.

To determine whether a method does, in fact, generate evidence, we need to be able to replicate it independently. But the premises imply that we don't have enough information to do so. Because we cannot independently verify which hypotheses the evidence generated by the method discriminates between, if any, there is no way to know what it means.

> **Comments:**
> **username_1**: First time I did a pure methodology answer, so call it a test. I don't feel too good about the lack of references for the evidence definition, but I don't even know where I'd start tracking them down. Thoughts, comments, downvotes welcome.  
> **username_2**: I tend to agree with the conclusion that a normative or absolute understanding of the test is impossible given the premises that are in the question. I think the best solution -- it's not a very good one -- is to collect the data yourself.  
> **username_0**: Thanks for coming back to this question. To clarify, I didn't mean to imply in the question that item information was not available anywhere. I guess the question is referring to the general situation where there probably are previous uses of the measure in the literature, and there might be a key article that describes the measure somewhere. I guess I'm thinking of a good general strategy for securing relevant information and then using it to form an understanding of the scale properties.  
> **username_1**: @JeromyAnglim So, just to clarify ... something like (1) What information do we need to interpret a scale? (2) How can we most effectively get that information?  
> **username_0**: @ChristianHummeluhr Yep :-)  

---

