<p>Many <a href="https://en.wikipedia.org/wiki/Construct_%28philosophy%29" rel="nofollow noreferrer">construct</a> scales are developed with relatively little attention to the content.  Researchers may start with a large set of candidate questions, versions, formats, and phrasings, and then through a <a href="https://en.wikipedia.org/wiki/Construct_validity" rel="nofollow noreferrer">validation</a> process, narrow them down to a subset that correlates well with the construct that they are trying to measure.  The goal is a balance between keeping the test short while also retaining its predictive value.  In <a href="https://www.ncbi.nlm.nih.gov/pubmed/11439754" rel="nofollow noreferrer">Baron-Cohen et al's (2001)</a> introduction, they note:</p>

<blockquote>
  <p>The AQ as shown in the Appendix is the outcome of piloting multiple
  versions, over several years.</p>
</blockquote>

<p>In many cases, researchers care less about <em>why</em> the questions work, and more just <em>that</em> they work.</p>

<p>Having said that, the decision to go with a <a href="https://en.wikipedia.org/wiki/Ipsative" rel="nofollow noreferrer">forced-choice</a> format vs. say, a <a href="https://en.wikipedia.org/wiki/Likert_scale" rel="nofollow noreferrer">Likert-type</a> rating scale with more granularity and a neutral option, can be political:</p>

<blockquote>
  <p>Ipsative measures may be more useful for evaluating traits within an
  individual, whereas Likert-type scales are more useful for evaluating
  traits across individuals.</p>
</blockquote>

<p>This can make forced-choice measures desirable for stigmatized labels to discourage comparing individuals to one another:</p>

<blockquote>
  <p>Encouraging pupils to beat their previous scores can take peer
  pressure out of situations and eliminates the competitive element
  associated with norm-based referencing.</p>
</blockquote>

<p>Note also that the original study confirms that mathematicians score significantly higher on this scale, as is to be expected:</p>

<blockquote>
  <p>... scientists score higher than nonscientists; and within the
  sciences, mathematics, physical scientists, computer scientists, and
  engineers score higher than the more human or life-centered sciences
  of medicine (including veterinary science) and biology. This latter
  finding replicates our earlier studies finding a link between autism
  spectrum conditions and occupations/skills in maths, physics, and
  engineering.</p>
</blockquote>

<p>This suggests that the authors thought it perfectly valid for mathematicians to score higher on their autism scale (ie, this scale is not designed to differentiate the autistic tendencies of mathematicians from those of individuals diagnosed with autism).</p>

CC BY-SA 3.0
2017-02-20T11:25:31.460Z
Unless I completely misunderstood the terminology, the AS test is not an ipsative test. Rather it is a Likert-scale test without a neutral option. I also fail to see what assessing pupils and beating previous scores has to do with the AS test.
Wrzlprmft
CC BY-SA 3.0
2017-02-20T11:35:09.677Z
*the original study confirms that mathematicians score significantly higher on this scale, as is to be expected* – Sure they do, but that does not address my doubts, which (somewhat exaggerated) are: The way the test is designed, small tendencies are amplified. For example, a mathematician would will slightly agree with an “autistic” option, because they are primed that they are somewhat more autistic than average. Therefore a high AS score for mathematicians is expected from the test design and does not tell us anything about reality.
Wrzlprmft
CC BY-SA 3.0
2017-02-20T17:00:33.560Z
@Wrzlprmft Yes, it is a forced-choice (ipsative) test, the authors themselves state this in the paper.  A Likert-scale would have different scoring for slightly vs strongly (dis)agree.  I was merely pointing out that scoring in such tests discourages comparisons between patients (compared to say, IQ tests), which may make them preferable for diagnostic tests that have potentially stigmatizing labels.  The authors do not explain their decision one way or the other.
Arnon Weinberg
CC BY-SA 3.0
2017-02-20T17:05:56.240Z
Re telling us about reality, this is the purpose of validation (a topic far too large for this post, but see http://cogsci.stackexchange.com/questions/9610/what-is-the-relation-between-measures-constructs-and-concepts for more detail).  The validation process confirmed that the amplification is both expected and realized, and does in fact tell us what we want to know from this scale about reality - ie, that mathematicians do have higher than average autistic tendencies.  It's possible that you think this scale should measure something different than what the authors think it should measure.
Arnon Weinberg
CC BY-SA 3.0
2017-02-20T19:29:19.623Z
Regarding the subject of validation: I think I understand this. My problem is rather that the test has properties that will make it succeed validation irrespective of the desired qualities. As a simple and blatant example, consider a world where 80 % of all mathematicians are autistic and vice versa and consider a test that simply asks the subject whether they are mathematicians. This test is obviously flawed, yet it will have pass a basic validation.
Wrzlprmft
CC BY-SA 3.0
2017-02-20T20:01:29.180Z
Yes, absolutely.  As I said, that is a discussion beyond the scope of this question (though the post I linked to does discuss this issue - for example, your assertion that autism and mathematics are separate constructs is just as contentious as a scale that measures mathematics when it purports to measure autism!).  Anyways, it sounds like you have a good grasp of the issues already.
Arnon Weinberg
CC BY-SA 3.0
2017-02-22T09:26:09.573Z
*beyond the scope of this question* – well, it is more or less my central question. However, I am open to the possibility that the answer to this question is that the test is actually flawed in this respect.
Wrzlprmft
CC BY-SA 3.0
2017-02-22T16:12:04.083Z
Your question was "What are the justifications for such a test design", not "Is this test design flawed".  Had your question been "Is this test design flawed", then it would likely have been Closed for being Primarily Opinion Based, which is outside the purview of this forum.  I suspect you think this scale should measure something different than what the authors think it should measure.
Arnon Weinberg
CC BY-SA 3.0
2017-02-22T17:07:55.933Z
You are right, “this test is flawed” was a bit going too far. Rather, it was: “was this possible effect taken into account?” Anyway, I am not complaining.
Wrzlprmft
CC BY-SA 3.0
2017-02-20T07:21:58.623Z
2017-02-20T21:13:44.250Z
2017-02-20T21:13:44.250Z
Arnon Weinberg
<p>In the <a href="//dx.doi.org/10.1023/A:1005653411471" rel="nofollow noreferrer">Autism-Spectrum Quotient test</a>, the subject is confronted with fifty statements such as:</p>
<blockquote>
<p>I don’t particularly enjoy reading fiction.</p>
<p>I prefer to do things the same way over and over again.</p>
</blockquote>
<p>and is asked whether they</p>
<ul>
<li>definitely agree,</li>
<li>slightly agree,</li>
<li>slightly disagree, or</li>
<li>definitely disagree</li>
</ul>
<p>with them. There is no neutral option.</p>
<p>The test is scored as follows: If the subject agrees with an “autistic” statement, they score a point. The same applies, if they disagree with a “non-autistic” statement. There is no distinction between <em>slightly</em> and <em>definitely</em> agreeing (or disagreeing).</p>
<p>I would expect that this test design (no neutral option, no distinction between slight and strong tendencies) leads to an amplification of small effects, such as interpretation of the question, slight tendencies, priming, chance, and so on. While I acknowledge that this effect may be desired in customer surveys and similar, it seems problematic to me in a test for the purposes of diagnosis and epidemology (such as this test). For example, I would expect that people who are already diagnosed with autism (and know this) or people with clichéically autistic interests like mathematics are more likely to tend for the more autistic option – even if their actual stance on the question in neutral.</p>
<p>Now, I am a layman and I acknowledge that designing such a test is not easy, as one has to account for several effects. Hence I wonder: <strong>What are the justifications for such a test design</strong>, in particular in light of its applications (diagnosis and epidemology) and my above criticism? The <a href="//dx.doi.org/10.1023/A:1005653411471" rel="nofollow noreferrer">paper introducing the test</a> does not provide any citations or reasons for this, at least not in those sections where I would expect it.</p>

CC BY-SA 3.0
2016-12-21T08:51:21.110Z
This is a totally valid, but super-broad question. There are 4 sub-questions to this one question. First, was is the history of the Autism-Spectrum Quotient test and how has it been validated/contested over the years. Second, how much are the types of tests affected by priming and stereotypes. Third, is this test undermined by a lack of neutral option. Fourth, are surveys without methods of comparison still valid. Please split this question into sub-questions and I will upvote them all.
Seanny123
CC BY-SA 3.0
2016-12-21T09:13:28.707Z
@Seanny123: I understand your point and given that I cannot even remotely estimate how long potential answers would be, you may very well be correct. However, before I follow your suggestion, please consider the following: 1) I am not much interested in the test’s history or validation per se, unless it provides answers to my main question(s). 2) I am not that much concerned about the issues addressed in your last three questions on their own (e.g., I fully acknowledge that the influence of priming and stereotypes is inevitable). Instead I am concerned in their interplay.
Wrzlprmft
CC BY-SA 3.0
2016-12-21T09:15:19.163Z
I understand that you're (legitimately) only interested in their interplay, but I guess what I'm claiming is that I personally think we have to first establish their individual validity before considering their interplay.
Seanny123
CC BY-SA 3.0
2016-12-21T13:30:49.390Z
@Seanny123: I narrowed down the question, though not as far as you suggested. I still hope that this is sufficiently narrow now.
Wrzlprmft
CC BY-SA 3.0
2017-02-08T07:56:16.420Z
I believe the question has been reduced in such a degree that it became answerable. Simply, why do the developers of the Autism-Spectrum Quotient test give four options, if they scale it down to just two (agree or disagree)?
Robin Kramer-ten Have
CC BY-SA 3.0
2017-03-01T09:36:16.333Z
@what: *Why do you assume that the high-scoring mathematicians do not have Asperger or autism?* – Where do say this? Just that I am concerned whether a test is flawed does not mean that I consider all conclusions drawn from its results wrong. My problem with the test design is (to be blunt) that it may measure whether somebody is a mathematician and not whether somebody is autistic. This would be problematic since the test may e.g. diagnose autism in a non-autistic mathematician.
Wrzlprmft
CC BY-SA 3.0
2017-03-01T16:54:12.987Z
@what: There is an underlying truth to many clichés, but that does not free them from being clichés. Not everybody interested in mathematics is automatically autistic. Still such a person may be primed that they are more “autistic” than the average population and thus be more likely to choose the “autistic” option in the test (because they are forced to make a choice). But such priming is something that you do not want to be reflected by a test (of course you cannot completely avoid it, but this test seems to amplify it).
Wrzlprmft
<abnormal-psychology><measurement><methodology><clinical-psychology><survey>
Rationale behind an “all-or-nothing” design for a test for the purposes of diagnosis and epidemology
Wrzlprmft
# Rationale behind an “all-or-nothing” design for a test for the purposes of diagnosis and epidemology

(Asked by: username_0 on 2016-12-20)

In the [Autism-Spectrum Quotient test](//dx.doi.org/10.1023/A:1005653411471), the subject is confronted with fifty statements such as:

> I don’t particularly enjoy reading fiction.
>
> I prefer to do things the same way over and over again.

and is asked whether they

* definitely agree,
* slightly agree,
* slightly disagree, or
* definitely disagree

with them. There is no neutral option.

The test is scored as follows: If the subject agrees with an “autistic” statement, they score a point. The same applies, if they disagree with a “non-autistic” statement. There is no distinction between *slightly* and *definitely* agreeing (or disagreeing).

I would expect that this test design (no neutral option, no distinction between slight and strong tendencies) leads to an amplification of small effects, such as interpretation of the question, slight tendencies, priming, chance, and so on. While I acknowledge that this effect may be desired in customer surveys and similar, it seems problematic to me in a test for the purposes of diagnosis and epidemology (such as this test). For example, I would expect that people who are already diagnosed with autism (and know this) or people with clichéically autistic interests like mathematics are more likely to tend for the more autistic option – even if their actual stance on the question in neutral.

Now, I am a layman and I acknowledge that designing such a test is not easy, as one has to account for several effects. Hence I wonder: **What are the justifications for such a test design**, in particular in light of its applications (diagnosis and epidemology) and my above criticism? The [paper introducing the test](//dx.doi.org/10.1023/A:1005653411471) does not provide any citations or reasons for this, at least not in those sections where I would expect it.

**Tags:** abnormal-psychology, measurement, methodology, clinical-psychology, survey

> **Comments:**
> **username_1**: This is a totally valid, but super-broad question. There are 4 sub-questions to this one question. First, was is the history of the Autism-Spectrum Quotient test and how has it been validated/contested over the years. Second, how much are the types of tests affected by priming and stereotypes. Third, is this test undermined by a lack of neutral option. Fourth, are surveys without methods of comparison still valid. Please split this question into sub-questions and I will upvote them all.  
> **username_0**: @username_1: I understand your point and given that I cannot even remotely estimate how long potential answers would be, you may very well be correct. However, before I follow your suggestion, please consider the following: 1) I am not much interested in the test’s history or validation per se, unless it provides answers to my main question(s). 2) I am not that much concerned about the issues addressed in your last three questions on their own (e.g., I fully acknowledge that the influence of priming and stereotypes is inevitable). Instead I am concerned in their interplay.  
> **username_1**: I understand that you're (legitimately) only interested in their interplay, but I guess what I'm claiming is that I personally think we have to first establish their individual validity before considering their interplay.  
> **username_0**: @username_1: I narrowed down the question, though not as far as you suggested. I still hope that this is sufficiently narrow now.  
> **username_2**: I believe the question has been reduced in such a degree that it became answerable. Simply, why do the developers of the Autism-Spectrum Quotient test give four options, if they scale it down to just two (agree or disagree)?  
> **username_0**: @what: *Why do you assume that the high-scoring mathematicians do not have Asperger or autism?* – Where do say this? Just that I am concerned whether a test is flawed does not mean that I consider all conclusions drawn from its results wrong. My problem with the test design is (to be blunt) that it may measure whether somebody is a mathematician and not whether somebody is autistic. This would be problematic since the test may e.g. diagnose autism in a non-autistic mathematician.  
> **username_0**: @what: There is an underlying truth to many clichés, but that does not free them from being clichés. Not everybody interested in mathematics is automatically autistic. Still such a person may be primed that they are more “autistic” than the average population and thus be more likely to choose the “autistic” option in the test (because they are forced to make a choice). But such priming is something that you do not want to be reflected by a test (of course you cannot completely avoid it, but this test seems to amplify it).  

---

## 1 answers

### Answer by username_3 on 2017-02-20. Score: 3 | Accepted answer

Many [construct](https://en.wikipedia.org/wiki/Construct_%28philosophy%29) scales are developed with relatively little attention to the content. Researchers may start with a large set of candidate questions, versions, formats, and phrasings, and then through a [validation](https://en.wikipedia.org/wiki/Construct_validity) process, narrow them down to a subset that correlates well with the construct that they are trying to measure. The goal is a balance between keeping the test short while also retaining its predictive value. In [Baron-Cohen et al's (2001)](https://www.ncbi.nlm.nih.gov/pubmed/11439754) introduction, they note:

> The AQ as shown in the Appendix is the outcome of piloting multiple
> versions, over several years.

In many cases, researchers care less about *why* the questions work, and more just *that* they work.

Having said that, the decision to go with a [forced-choice](https://en.wikipedia.org/wiki/Ipsative) format vs. say, a [Likert-type](https://en.wikipedia.org/wiki/Likert_scale) rating scale with more granularity and a neutral option, can be political:

> Ipsative measures may be more useful for evaluating traits within an
> individual, whereas Likert-type scales are more useful for evaluating
> traits across individuals.

This can make forced-choice measures desirable for stigmatized labels to discourage comparing individuals to one another:

> Encouraging pupils to beat their previous scores can take peer
> pressure out of situations and eliminates the competitive element
> associated with norm-based referencing.

Note also that the original study confirms that mathematicians score significantly higher on this scale, as is to be expected:

> ... scientists score higher than nonscientists; and within the
> sciences, mathematics, physical scientists, computer scientists, and
> engineers score higher than the more human or life-centered sciences
> of medicine (including veterinary science) and biology. This latter
> finding replicates our earlier studies finding a link between autism
> spectrum conditions and occupations/skills in maths, physics, and
> engineering.

This suggests that the authors thought it perfectly valid for mathematicians to score higher on their autism scale (ie, this scale is not designed to differentiate the autistic tendencies of mathematicians from those of individuals diagnosed with autism).

> **Comments:**
> **username_0**: Unless I completely misunderstood the terminology, the AS test is not an ipsative test. Rather it is a Likert-scale test without a neutral option. I also fail to see what assessing pupils and beating previous scores has to do with the AS test.  
> **username_0**: *the original study confirms that mathematicians score significantly higher on this scale, as is to be expected* – Sure they do, but that does not address my doubts, which (somewhat exaggerated) are: The way the test is designed, small tendencies are amplified. For example, a mathematician would will slightly agree with an “autistic” option, because they are primed that they are somewhat more autistic than average. Therefore a high AS score for mathematicians is expected from the test design and does not tell us anything about reality.  
> **username_3**: @username_0 Yes, it is a forced-choice (ipsative) test, the authors themselves state this in the paper.  A Likert-scale would have different scoring for slightly vs strongly (dis)agree.  I was merely pointing out that scoring in such tests discourages comparisons between patients (compared to say, IQ tests), which may make them preferable for diagnostic tests that have potentially stigmatizing labels.  The authors do not explain their decision one way or the other.  
> **username_3**: Re telling us about reality, this is the purpose of validation (a topic far too large for this post, but see http://cogsci.stackexchange.com/questions/9610/what-is-the-relation-between-measures-constructs-and-concepts for more detail).  The validation process confirmed that the amplification is both expected and realized, and does in fact tell us what we want to know from this scale about reality - ie, that mathematicians do have higher than average autistic tendencies.  It's possible that you think this scale should measure something different than what the authors think it should measure.  
> **username_0**: Regarding the subject of validation: I think I understand this. My problem is rather that the test has properties that will make it succeed validation irrespective of the desired qualities. As a simple and blatant example, consider a world where 80 % of all mathematicians are autistic and vice versa and consider a test that simply asks the subject whether they are mathematicians. This test is obviously flawed, yet it will have pass a basic validation.  
> **username_3**: Yes, absolutely.  As I said, that is a discussion beyond the scope of this question (though the post I linked to does discuss this issue - for example, your assertion that autism and mathematics are separate constructs is just as contentious as a scale that measures mathematics when it purports to measure autism!).  Anyways, it sounds like you have a good grasp of the issues already.  
> **username_0**: *beyond the scope of this question* – well, it is more or less my central question. However, I am open to the possibility that the answer to this question is that the test is actually flawed in this respect.  
> **username_3**: Your question was "What are the justifications for such a test design", not "Is this test design flawed".  Had your question been "Is this test design flawed", then it would likely have been Closed for being Primarily Opinion Based, which is outside the purview of this forum.  I suspect you think this scale should measure something different than what the authors think it should measure.  
> **username_0**: You are right, “this test is flawed” was a bit going too far. Rather, it was: “was this possible effect taken into account?” Anyway, I am not complaining.  

---

