<p><sub>Self-report methodology was one of my qualifying exam topics as a doctoral student of social and personality psychology, so I've got a ton of references to offer, but I confess I haven't read most of them very thoroughly (if at all), and I've forgotten where exactly I've read some of this. It's really a very broad topic as well, so I won't list most I know of that might be relevant here, but I can probably offer more on specific subtopics per request. </sub></p>

<p>Very many relationships exist between respondent characteristics and response biases that depend on survey design. It would be practically unimaginable if topic-in-question weren't a moderating factor for some of them. <a href="https://en.wikipedia.org/wiki/Social_desirability_bias" rel="nofollow">Social desirability</a> in particular is a form of bias that depends heavily on survey design, participant characteristics, and topic of study. Wikipedia offers a pretty good intro-level summary:</p>

<blockquote>
  <p>The tendency poses a serious problem with conducting research with <strong>self-reports, especially questionnaires</strong>. This bias interferes with the interpretation of average tendencies as well as individual differences.</p>
  
  <p><strong>Topics</strong> where socially desirable responding (SDR) is of special concern are self-reports of <strong>abilities, personality, sexual behavior, and drug use</strong>. When confronted with the question "How often do you masturbate?", for example, respondents may be pressured by the societal taboo against masturbation, and either under-report the frequency or avoid answering the question. Therefore the mean rates of masturbation derived from self-report surveys are likely to be severe underestimates.</p>
  
  <p>When confronted with the question, "Do you use drugs/<strong>illicit</strong> substances?" the respondent may be influenced by the fact that controlled substances, including the more commonly used marijuana, are generally <strong>illegal</strong>. Respondents may feel pressured to deny any drug use or rationalize it, e.g., "I only smoke marijuana when my friends are around." The bias can also influence reports of <strong>number of sexual partners</strong>. In fact, the <strong>bias may operate in opposite directions for different subgroups</strong>: Whereas men tend to inflate the numbers, women tend to underestimate theirs. In either case, the mean reports from both groups are likely to be distorted by social desirability bias.</p>
  
  <p><strong>Other topics that are sensitive to social desirability bias:</strong></p>
  
  <ul>
  <li>Personal income and earnings, often inflated when low and deflated when high.</li>
  <li>Feelings of low self-worth and/or powerlessness, often denied.</li>
  <li>Excretory functions, often approached uncomfortably, if discussed at all.</li>
  <li>Compliance with medicinal dosing schedules, often inflated.</li>
  <li>Religion, often either avoided or uncomfortably approached.</li>
  <li>Patriotism, either inflated or, if denied, done so with a fear of other party's judgement.</li>
  <li>Bigotry and intolerance, often denied, even if it exists within the responder.</li>
  <li>Intellectual achievements, often inflated.</li>
  <li>Physical appearance, either inflated or deflated</li>
  <li>Acts of real or imagined physical violence, often denied.</li>
  <li>Indicators of charity or "benevolence," often inflated.</li>
  <li>Illegal acts, often denied.
  <br>[Emphasis added.]</li>
  </ul>
</blockquote>

<p>An arguable exception to the second-to-last appears in Smith (1987): people respond less generously and more negatively to "welfare" than "poor".</p>

<p>Wikipedia mentions one of many measures of socially desirable responding, but notes:</p>

<blockquote>
  <p>The major concern with SDR scales is that they confound style with content. After all, people actually differ in the degree to which they possess desirable traits (e.g., nuns versus criminals).</p>
</blockquote>

<p>Another concern is the potential multidimensionality of social desirability bias. Some arises in consciously deceptive responding; another kind arises from unconscious self-deception. These extent of each of these kinds of social desirability bias may depend differently on individual differences in respondents and question topic and wording.</p>

<p>Wikipedia also gives a brief overview of <a href="https://en.wikipedia.org/wiki/Social_desirability_bias#Other_response_styles" rel="nofollow">other response styles</a>, including extreme response style, which relates to demographic (Song, 2007) and personality (Hamilton, 1968) factors. An opposite sort of style, the tendency toward neutral / "don't know" responses may be caused by personal uncertainty, item difficulty, or the sensitivity of the topic (Coombs &amp; Coombs, 1976), so this seems to be a pretty complete example of a kind of bias that depends on all three factors. Another one of these response styles, acquiescence (and vice versa), is a somewhat interesting personality variable unto itself (Couch &amp; Keniston, 1960); its influence can be limited by avoiding the "agree vs. disagree" format, but this may be harder in attitude research than in some other topics.</p>

<p>Below this, Wikipedia mentions that anonymity, confidentiality, and neutralized administration can reduce social desirability bias, so in this sense, contextual factors outside the question itself can affect biasing, especially where the judgment of others (e.g., parents, peers, employers, law enforcement) is of concern (e.g., children of authoritarian parents, friends of judgmental people, job applicants facing personality tests, parolees and prior convicts).</p>

<p>A particularly good review article I have on hand (<a href="https://edit.ethz.ch/oat/education/material/material_06_07/material_empirical_methods_06_07/American_Psychologist_Self-Reports.pdf" rel="nofollow">Schwarz, 1999</a>) discusses issues of "question wording, format, and context," including:</p>

<ul>
<li>Ambiguity, obscurity, and wordiness</li>
<li>Open-ended responding (see also Geer, 1988) vs. frequency reporting and item rating</li>
<li>Topical subjectivity, demand for expertise, and time-dependence</li>
</ul>

<p>As this article is a review itself, you can also find plenty of other references through it. Another review article that might interest others here relates survey measurement issues to cognitive issues like memory, comprehension, and schema priming (Jobe &amp; Mingay, 1991), each of which I could easily see being more or less relevant in certain contexts and for certain people in ways that would depend on how a question is worded. Again, another huge reference section to be mined for more specifics there too. Yet another review (<a href="http://www.apologeticsinthechurch.com/uploads/7/4/5/6/7456646/nisbettwilson.pdf" rel="nofollow">Nisbett &amp; Wilson, 1977</a>) pertains specifically to error (not necessarily systematic bias) in self-reported information about mental processes due to limited self-awareness / insight and the influence of prior theories or plausibility judgments. Good stuff to bring up here, if I do say so myself!</p>

<p><strong>References</strong></p>

<p>Coombs, C. H., &amp; Coombs, L. C. (1976). “Don't Know” Item Ambiguity or Respondent Uncertainty?. <em>Public Opinion Quarterly, 40</em>(4), 497–514.</p>

<p>Couch, A., &amp; Keniston, K. (1960). Yeasayers and naysayers: Agreeing response set as a personality variable. <em>Journal of Abnormal and Social Psychology, 60</em>(2), 151–174.</p>

<p>Geer, J. G. (1988). What do open-ended questions measure? <em>Public Opinion Quarterly, 52</em>(3), 365–367.</p>

<p>Hamilton, D.L. (1968). Personality attributes associated with extreme response style. <em>Psychological Bulletin, 69</em>(3), 192–203.</p>

<p>Jobe, J. B., &amp; Mingay, D. J. (1991). Cognition and survey measurement: History and overview. <em>Applied Cognitive Psychology, 5</em>(3), 175–192.</p>

<p>Nisbett, R. E., &amp; Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. <em>Psychological Review, 84</em>(3), 231–259. Available online, URL: <a href="http://www.apologeticsinthechurch.com/uploads/7/4/5/6/7456646/nisbettwilson.pdf" rel="nofollow">http://www.apologeticsinthechurch.com/uploads/7/4/5/6/7456646/nisbettwilson.pdf</a>. Retrieved February 5, 2014.</p>

<p>Schwarz, N. (1999). Self-reports: How the questions shape the answers. <em>American Psychologist, 54</em>(2), 93–105. Available online, URL: <a href="https://edit.ethz.ch/oat/education/material/material_06_07/material_empirical_methods_06_07/American_Psychologist_Self-Reports.pdf" rel="nofollow">https://edit.ethz.ch/oat/education/material/material_06_07/material_empirical_methods_06_07/American_Psychologist_Self-Reports.pdf</a>.</p>

<p>Smith, T. W. (1987). That which we call welfare by any other name would smell sweeter: An analysis of the impact of question wording on response patterns. <em>Public Opinion Quarterly, 51</em>(1), 75–83.</p>

<p>Song, X.-Y. (2007). Analysis of multisample structural equation models with applications to Quality of Life data. In S.-Y. Lee (Ed.), <em>Handbook of latent variable and related models</em>, pp. 279–302. North-Holland.</p>

CC BY-SA 3.0
2014-02-06T04:25:08.490Z
2014-02-08T21:22:45.103Z
2014-02-08T21:22:45.103Z
Nick Stauner
<p>I just read that in a survey on legalizing abortions:</p>

<blockquote>
  <p>Responses depended strongly on the question wording. </p>
</blockquote>

<p>But I have also seen cases where question wording is largely irrelevant. As long as there are no intentional "leading" phrases or general errors like ambiguity, respondents are likely to respond in a similar manner to questions with a similar content. </p>

<p>My hunch, based on personal experience in non-survey contexts, would be that in issues where people have already formed strong attitudes on the topic, a wording which is congruent with existing attitudes will elicit more agreement, and a wording which implies disagreement will elicit more disagreement. But in cases where people don't already have attitudes, the question wording won't be that important. But this is really just a hunch, I have never seen data on that. </p>

<p>Is there a good scientific overview on the factors which determine the relationship between the question topic, the respondent (whether his attitudes or other characteristics), and the likelihood that a difference in question wording will cause a difference in response? </p>

CC BY-SA 3.0
2014-02-05T21:02:59.723Z
Most of the tags I thought of were not present on the site: [psychometry] and [confounding-variables], for example. If this type of question is ontopic, maybe somebody can create them.
rumtscho
CC BY-SA 3.0
2014-02-05T23:10:07.330Z
"psychometrics" is a synonym for [tag:measurement], but "psychometry" would be a good alternate synonym to add. Your question is certainly on-topic here! I've added some existing tags to help classify it.
Nick Stauner
<measurement><language><bias><survey>
In measuring preferences, is the magnitude of the question wording effect topic-dependent?
rumtscho
# In measuring preferences, is the magnitude of the question wording effect topic-dependent?

(Asked by: username_0 on 2014-02-05)

I just read that in a survey on legalizing abortions:

> Responses depended strongly on the question wording.

But I have also seen cases where question wording is largely irrelevant. As long as there are no intentional "leading" phrases or general errors like ambiguity, respondents are likely to respond in a similar manner to questions with a similar content.

My hunch, based on personal experience in non-survey contexts, would be that in issues where people have already formed strong attitudes on the topic, a wording which is congruent with existing attitudes will elicit more agreement, and a wording which implies disagreement will elicit more disagreement. But in cases where people don't already have attitudes, the question wording won't be that important. But this is really just a hunch, I have never seen data on that.

Is there a good scientific overview on the factors which determine the relationship between the question topic, the respondent (whether his attitudes or other characteristics), and the likelihood that a difference in question wording will cause a difference in response?

**Tags:** measurement, language, bias, survey

> **Comments:**
> **username_0**: Most of the tags I thought of were not present on the site: [psychometry] and [confounding-variables], for example. If this type of question is ontopic, maybe somebody can create them.  
> **username_1**: "psychometrics" is a synonym for [tag:measurement], but "psychometry" would be a good alternate synonym to add. Your question is certainly on-topic here! I've added some existing tags to help classify it.  

---

## 1 answers

### Answer by username_1 on 2014-02-06. Score: 3

Self-report methodology was one of my qualifying exam topics as a doctoral student of social and personality psychology, so I've got a ton of references to offer, but I confess I haven't read most of them very thoroughly (if at all), and I've forgotten where exactly I've read some of this. It's really a very broad topic as well, so I won't list most I know of that might be relevant here, but I can probably offer more on specific subtopics per request.

Very many relationships exist between respondent characteristics and response biases that depend on survey design. It would be practically unimaginable if topic-in-question weren't a moderating factor for some of them. [Social desirability](https://en.wikipedia.org/wiki/Social_desirability_bias) in particular is a form of bias that depends heavily on survey design, participant characteristics, and topic of study. Wikipedia offers a pretty good intro-level summary:

> The tendency poses a serious problem with conducting research with **self-reports, especially questionnaires**. This bias interferes with the interpretation of average tendencies as well as individual differences.
>
> **Topics** where socially desirable responding (SDR) is of special concern are self-reports of **abilities, personality, sexual behavior, and drug use**. When confronted with the question "How often do you masturbate?", for example, respondents may be pressured by the societal taboo against masturbation, and either under-report the frequency or avoid answering the question. Therefore the mean rates of masturbation derived from self-report surveys are likely to be severe underestimates.
>
> When confronted with the question, "Do you use drugs/**illicit** substances?" the respondent may be influenced by the fact that controlled substances, including the more commonly used marijuana, are generally **illegal**. Respondents may feel pressured to deny any drug use or rationalize it, e.g., "I only smoke marijuana when my friends are around." The bias can also influence reports of **number of sexual partners**. In fact, the **bias may operate in opposite directions for different subgroups**: Whereas men tend to inflate the numbers, women tend to underestimate theirs. In either case, the mean reports from both groups are likely to be distorted by social desirability bias.
>
> **Other topics that are sensitive to social desirability bias:**
>
> * Personal income and earnings, often inflated when low and deflated when high.
> * Feelings of low self-worth and/or powerlessness, often denied.
> * Excretory functions, often approached uncomfortably, if discussed at all.
> * Compliance with medicinal dosing schedules, often inflated.
> * Religion, often either avoided or uncomfortably approached.
> * Patriotism, either inflated or, if denied, done so with a fear of other party's judgement.
> * Bigotry and intolerance, often denied, even if it exists within the responder.
> * Intellectual achievements, often inflated.
> * Physical appearance, either inflated or deflated
> * Acts of real or imagined physical violence, often denied.
> * Indicators of charity or "benevolence," often inflated.
> * Illegal acts, often denied.
>     
>   [Emphasis added.]

An arguable exception to the second-to-last appears in Smith (1987): people respond less generously and more negatively to "welfare" than "poor".

Wikipedia mentions one of many measures of socially desirable responding, but notes:

> The major concern with SDR scales is that they confound style with content. After all, people actually differ in the degree to which they possess desirable traits (e.g., nuns versus criminals).

Another concern is the potential multidimensionality of social desirability bias. Some arises in consciously deceptive responding; another kind arises from unconscious self-deception. These extent of each of these kinds of social desirability bias may depend differently on individual differences in respondents and question topic and wording.

Wikipedia also gives a brief overview of [other response styles](https://en.wikipedia.org/wiki/Social_desirability_bias#Other_response_styles), including extreme response style, which relates to demographic (Song, 2007) and personality (Hamilton, 1968) factors. An opposite sort of style, the tendency toward neutral / "don't know" responses may be caused by personal uncertainty, item difficulty, or the sensitivity of the topic (Coombs & Coombs, 1976), so this seems to be a pretty complete example of a kind of bias that depends on all three factors. Another one of these response styles, acquiescence (and vice versa), is a somewhat interesting personality variable unto itself (Couch & Keniston, 1960); its influence can be limited by avoiding the "agree vs. disagree" format, but this may be harder in attitude research than in some other topics.

Below this, Wikipedia mentions that anonymity, confidentiality, and neutralized administration can reduce social desirability bias, so in this sense, contextual factors outside the question itself can affect biasing, especially where the judgment of others (e.g., parents, peers, employers, law enforcement) is of concern (e.g., children of authoritarian parents, friends of judgmental people, job applicants facing personality tests, parolees and prior convicts).

A particularly good review article I have on hand ([Schwarz, 1999](https://edit.ethz.ch/oat/education/material/material_06_07/material_empirical_methods_06_07/American_Psychologist_Self-Reports.pdf)) discusses issues of "question wording, format, and context," including:

* Ambiguity, obscurity, and wordiness
* Open-ended responding (see also Geer, 1988) vs. frequency reporting and item rating
* Topical subjectivity, demand for expertise, and time-dependence

As this article is a review itself, you can also find plenty of other references through it. Another review article that might interest others here relates survey measurement issues to cognitive issues like memory, comprehension, and schema priming (Jobe & Mingay, 1991), each of which I could easily see being more or less relevant in certain contexts and for certain people in ways that would depend on how a question is worded. Again, another huge reference section to be mined for more specifics there too. Yet another review ([Nisbett & Wilson, 1977](http://www.apologeticsinthechurch.com/uploads/7/4/5/6/7456646/nisbettwilson.pdf)) pertains specifically to error (not necessarily systematic bias) in self-reported information about mental processes due to limited self-awareness / insight and the influence of prior theories or plausibility judgments. Good stuff to bring up here, if I do say so myself!

**References**

Coombs, C. H., & Coombs, L. C. (1976). “Don't Know” Item Ambiguity or Respondent Uncertainty?. *Public Opinion Quarterly, 40*(4), 497–514.

Couch, A., & Keniston, K. (1960). Yeasayers and naysayers: Agreeing response set as a personality variable. *Journal of Abnormal and Social Psychology, 60*(2), 151–174.

Geer, J. G. (1988). What do open-ended questions measure? *Public Opinion Quarterly, 52*(3), 365–367.

Hamilton, D.L. (1968). Personality attributes associated with extreme response style. *Psychological Bulletin, 69*(3), 192–203.

Jobe, J. B., & Mingay, D. J. (1991). Cognition and survey measurement: History and overview. *Applied Cognitive Psychology, 5*(3), 175–192.

Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. *Psychological Review, 84*(3), 231–259. Available online, URL: <http://www.apologeticsinthechurch.com/uploads/7/4/5/6/7456646/nisbettwilson.pdf>. Retrieved February 5, 2014.

Schwarz, N. (1999). Self-reports: How the questions shape the answers. *American Psychologist, 54*(2), 93–105. Available online, URL: <https://edit.ethz.ch/oat/education/material/material_06_07/material_empirical_methods_06_07/American_Psychologist_Self-Reports.pdf>.

Smith, T. W. (1987). That which we call welfare by any other name would smell sweeter: An analysis of the impact of question wording on response patterns. *Public Opinion Quarterly, 51*(1), 75–83.

Song, X.-Y. (2007). Analysis of multisample structural equation models with applications to Quality of Life data. In S.-Y. Lee (Ed.), *Handbook of latent variable and related models*, pp. 279–302. North-Holland.


---

