<h1>Intro</h1>

<p>Here's a slightly more compromising (yet arguably more demanding) answer than what's, which is probably a good representative of how the idea of changing items will generally be received by reviewers, supervisors, or substantive theorists. From the perspective of a methodology nerd (don't mean to deny @what that dubious distinction too BTW), there are some reasons to consider reworking this questionnaire and certainly very many other measures in common usage. It's true that this changes the latent construct and threatens its comparability to that estimated in previous research, but changes may be justified, and comparability may be worth sacrificing or even entirely undesirable if the measurement model in previous research was just that badly invalid.</p>

<p>It's also true that there is a lot of sophisticated, challenging, expensive work to do in the construct validation process, but as such, much of this work never gets done. For both theoretical and practical reasons, as much of this work as is feasible should be completed, reported, and kept up-to-date, and it's not necessarily any less important to do when one isn't motivated to change the measure. Building on a long line of previous research by using old methods mostly protects comparability of results within that line of research; it doesn't protect the line of research as a whole from its systemic flaws. Enough long lines of heralded theory have fallen over time to discourage complacency based on unquestioning deference to seemingly established methods.</p>

<h1>Scrutiny of <a href="http://gip.uniovi.es/gdiyad/docume/arti_jep01.pdf" rel="nofollow noreferrer">Milfont &amp; Duckitt (2004)</a></h1>

<h2><em>The question in question is questionable.</em></h2>

<p>To demonstrate the methodological vulnerabilities of even relatively modern research, I'll nitpick your specific example. First, as you've noted, the focal question itself is <a href="https://en.wikipedia.org/wiki/Double-barreled_question" rel="nofollow noreferrer"><strong>double- or even triple-barreled</strong></a>:</p>

<ol>
<li>Should weeds be eradicated?</li>
<li>Do weeds inhibit the full development of <em>useful</em> plants?

<ul>
<li>Do weeds inhibit the full development of <em>ornamental</em> plants?</li>
</ul></li>
</ol>

<p>This decreases interpretability of results somewhat needlessly (the measure is long enough already that one extra question wouldn't hurt noticeably...and what good does the distinction between useful and ornamental plants do?) and probably increases measurement error, as respondents are likely to rate their agreement somewhat haphazardly based on whichever part of the question elicits the strongest attitudinal response.</p>

<p>Second, this particular item loads rather weakly ($\lambda=.38)$ on a factor (the tenth and least reliable factor, $\alpha=.58$) with no negatively-weighted items. <a href="http://gip.uniovi.es/gdiyad/docume/arti_jep01.pdf" rel="nofollow noreferrer">Milfont and Duckitt <sup>(2004)</sup></a> critique another measure for possessing unbalanced subscales, which by their own admission are "open to <a href="https://en.wikipedia.org/wiki/Acquiescence_bias" rel="nofollow noreferrer"><strong>acquiescence bias</strong></a>". Yet they haven't fully balanced their own subscales, including also those for factors 2, 4, and 6 – this is just one shy of the number of unbalanced subscales in the measure they critique!</p>

<p>Regardless, "balancing" subscale items doesn't eliminate acquiescence bias; it only "balances" its effects on latent factor scores to the extent that they are calculated based on an equal number of equally-and-oppositely weighted items. In the authors' own usage of the measure (the structural equation model in Figure 1, p. 299), it appears that factor scores were based on the items weighted by their factor loadings, which are not equal. Hence even the factors that do have equal numbers of oppositely-weighted items (of which there are only two: factors 3 and 8) do not truly balance the effects of acquiescence bias.</p>

<p>Furthermore, even the false ideal of a set of items with equal, opposite, and balanced weighting would not truly balance acquiescence effects, as these also depend on item content and respondents' item-specific expertise <sup>(McClendon, 1991a)</sup>, both of which vary across items, of course. In the case of our double/triple-barreled question, the second/third barrel particularly concerns horticultural knowledge, so this item seems likely to suffer specially from acquiescence bias due to that second/third barrel that arguably does more harm than good. Personally, I'm somewhat surprised to find theoretical basis for suspecting that this expertise issue with one (or two) particular barrel(s) causes more acquiescence problems than the ambiguity of the question overall <sup>(McClendon, 1991b)</sup>, as is due to its compound nature. Acquiescence may be measurable and controllable <sup>(Billiet &amp; McClendon, 2000)</sup>, but it's not always a big threat, it's not the only response-style-based threat <sup>(<a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2736523/" rel="nofollow noreferrer">Hinz, Michalski, Schwarz, &amp; Herzberg, 2007</a>)</sup>, and balancing oppositely-weighted items halfheartedly is surely an incomplete amelioration <sup>(Schriesheim &amp; Hill, 1981)</sup>.</p>

<h2><em>The model is not a model model.</em></h2>

<p>This leads to further concerns with the overall measurement model. It seems to have been developed with methods that are somewhat <strong>inappropriate for Likert rating data</strong>, which are technically ordinal (i.e., discrete and ranked). The authors don't specify how exactly they handled item scoring, so they probably gave it the simple, conventional treatment as continuous data. This is not altogether indefensible with a seven-point Likert scale (see <a href="https://stats.stackexchange.com/a/86959/32036">this answer on Cross Validated</a>), but it's not ideal. The particular item in question appears on a scale with only four items, which is probably insufficient for producing a scale score by simple summation or averaging according to <a href="http://en.wikipedia.org/wiki/Classical_test_theory" rel="nofollow noreferrer">classical test theory (CTT)</a>; this may further explain the scale's relatively dismal $\alpha$ reliability. Nonetheless, the authors probably used some such CTT method to calculate the scale scores they tested for gender differences in their Table 4, as they haven't referred to these as factor scores, much less specified which kind of factor scores they might be (see <a href="http://pareonline.net/pdf/v14n20.pdf" rel="nofollow noreferrer">DiStefano, Zhu, &amp; Mîndrilă, <sup>(2009)</sup></a> for a review of six popular options).</p>

<p>Moreover, the scale scoring system appears to have been developed through application of principal components and principal axis factor extraction from a simple covariance matrix that probably treated the data as continuous, as well as maximum likelihood estimation of a confirmatory factor analytic (CFA) model's fit to that same matrix. See <a href="https://stats.stackexchange.com/q/2374/32036">this question on Cross Validated, "Factor analysis of questionnaires composed of Likert items"</a> for discussion of preferable methods and other unaddressed issues, including:</p>

<ul>
<li>Other estimation algorithms that are less biased by ordinal data than maximum likelihood</li>
<li>Fitting latent factor models to a polychoric correlation matrix, not a covariance matrix</li>
<li>Exploration of bifactor structure in the case of a strong general factor amidst multiple dimensions

<ul>
<li>This may be a particularly important alternative to consider alongside a second-order model.</li>
</ul></li>
<li>Preemptive control of various biases, including extreme response style, as via unfolding models</li>
</ul>

<p>It's also a bit fishy to perform exploratory factor analysis on the same sample as a CFA (see <a href="https://stats.stackexchange.com/q/83512/32036">yet another question on Cross Validated, "Regression testing after dimension reduction"</a>), though at least Milfont and Duckitt <sup>(2004)</sup> did this in the less objectionable order to confirm and explore somewhat distinct latent structures. In sum, reasons such as these may explain the authors' failure to achieve good model fit by the conventional standards they cited themselves. When model fit is inadequate, path coefficients may be biased, which may pose several further problems for the proposed latent factor structure.</p>

<h1>General reflection on alternative priorities</h1>

<p>In the end, problems such as these vary widely in consequentiality depending on the egregiousness of particular instances. Older measures probably generally suffer more for their age, as many of the methodological issues I've raised here arise from relatively modern research. Unfortunately, this may threaten the validity of retrospective comparisons most where they might otherwise be most richly informative – where lines of research reach back across several decades and even generations – but other sociological factors (e.g., <a href="https://en.wikipedia.org/wiki/Cohort_effect" rel="nofollow noreferrer">cohort effects</a>) may limit the equivalence of old research to new research anyway. Reasons such as these to give up on traditional methods are at least as numerous and important as reasons to adhere loyally.</p>

<p>In more modern instances such as <a href="http://gip.uniovi.es/gdiyad/docume/arti_jep01.pdf" rel="nofollow noreferrer">Milfont and Duckitt <sup>(2004)</sup></a>, there may be less to lose by parting ways with less established and replicated methods (though I see <a href="http://scholar.google.com/scholar?cites=14421768210628985778&amp;as_sdt=2005&amp;sciodt=0,5&amp;hl=en" rel="nofollow noreferrer">their article has been cited at least 180 times already</a>), but there may also be less to gain. <strong>Some of these issues I've raised may be fairly inconsequential, or if resolved, might even absolve their theory</strong> by improving model fit and demonstrating a lack of cause for further concern. One simply can't be sure without trying. </p>

<p>How much of this effort is worthwhile is an open question, a balancing act, a fundamental career choice, and even a bit of a gamble. Deep, unresolved, and even unresolvable epistemic problems underlie every application of empirical scientific methodology – this may be especially true of the social sciences – but time spent on basic research is not generally time spent directly on application of theory to contemporarily exigent problems. Thus one confronts the <strong>fundamental dilemma of addressing "real-world problems" now vs. theoretical problems for future applications</strong>, and through it the career choice of applied vs. theoretical and methodological research.</p>

<p>One way to strike a balance is to prognosticate the societal impact of your options, and in any given choice to go with the option most likely to matter. If your methodological problem seems more pragmatically consequential than your applied problem, solve it first, or instead if necessary! (This I've stated as an idealized, no-brainer platitude, but intended to reflect a subtler judgment call.) Another is to contribute in the ways you value most – the ways most motivating for you, be they immediate career advancement or long-term legacy – as service to your own values is service you're most likely to perform best and most valuably to all beneficiaries (yourself included). Another is to consider your implicit motives and core strengths: <strong>what do you like to do, and what do you do best?</strong> These factors will affect the value of your work too, and should inform your choices of basic priorities and specific endeavors.</p>

<p>Hopefully you can find ways to make these priorities align in your work, but everyone's bound to face conflicts. I'm working through some of these conflicts presently myself – hence the indulgence in the preceding exegesis – but I'll spare you further autobiography. Ponder instead <a href="https://www.youtube.com/watch?v=txWWi2n76FI" rel="nofollow noreferrer">the wisdom of Peppy Hare</a>, and let me know if you've got some data and want to collaborate on updating your measurement models!</p>

<p><strong>References</strong></p>

<p><sup>
- Billiet, J. B., &amp; McClendon, M. J. (2000). Modeling acquiescence in measurement models for two balanced sets of items. <em>Structural Equation Modeling, 7</em>(4), 608–628.<br>
- DiStefano, C., Zhu, M., &amp; Mîndrilă, D. (2009). Understanding and using factor scores: Considerations for the applied researcher. <em>Practical Assessment, Research &amp; Evaluation, 14</em>(20), 1–11. Retrieved from <a href="http://pareonline.net/pdf/v14n20.pdf" rel="nofollow noreferrer">http://pareonline.net/pdf/v14n20.pdf</a>.<br>
- Hinz, A., Michalski, D., Schwarz, R., &amp; Herzberg, P. Y. (2007). The acquiescence effect in responding to a questionnaire. <em>GMS Psycho-Social Medicine, 4</em>. Retrieved from <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2736523/" rel="nofollow noreferrer">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2736523/</a>.<br>
- McClendon, M. J. (1991a). Acquiescence and recency response-order effects in interview surveys. <em>Sociological Methods &amp; Research, 20</em>(1), 60–103.<br>
- McClendon, M. J. (1991b). Acquiescence: Tests of the cognitive limitations and question ambiguity hypotheses. <em>Journal of Official Statistics, 7</em>(2), 153–166.<br>
- Milfont, T. L., &amp; Duckitt, J. (2004). The structure of environmental attitudes: A first- and second-order confirmatory factor analysis. <em>Journal of Environmental Psychology, 24</em>, 289–303. Retrieved from <a href="http://gip.uniovi.es/gdiyad/docume/arti_jep01.pdf" rel="nofollow noreferrer">http://gip.uniovi.es/gdiyad/docume/arti_jep01.pdf</a>.<br>
- Schriesheim, C. A., &amp; Hill, K. D. (1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. <em>Educational and Psychological Measurement, 41</em>(4), 1101–1114.<br>
</sup></p>

CC BY-SA 3.0
2014-02-24T12:58:23.467Z
This is an excellent writeup. Thank you!
Adb
CC BY-SA 3.0
2014-02-24T15:49:15.110Z
Glad it was helpful!
Nick Stauner
CC BY-SA 3.0
2014-02-21T02:14:04.267Z
2014-02-21T02:14:04.267Z
2017-04-13T12:44:18.443Z
Nick Stauner
<p>To reiterate @what's answer, here are a few simple rules.</p>

<h3>Change wording when...</h3>

<ul>
<li>You want to improve the scale and are prepared to engage in a thorough validation process of the new scale. E.g., comparing the new with the old version; re-running factor analyses, reliability analyses, validation. Naturally you'd only do this if you felt that you could make a substantial improvement to the scale.</li>
<li>You need to substantially adapt a scale to your context. E.g., the test was looking at attitudes to X and you want to look at attitudes to Y. You only have room in your scale for a subset of the items. In all these cases, you are only loosely drawing on the existing validity information for the scale. And adapting a scale to your context should not be your first choice if there is an existing scale already suited to your context.</li>
</ul>

<h3>Don't change wording when...</h3>

<ul>
<li>You think that minor changes would make the scale better (e.g., by adding/removing an item; tweaking the wording). Such changes prevent you from comparing your data to other normative studies. While if the changes are minor, the changes to the psychometrics are also likely to be minor, this is ultimately an empirical question. And thus, you are limited in drawing on the psychometric evidence of other studies after you have made changes. Thus, it is rarely worth it.</li>
</ul>

<h3>Thoughts on changes</h3>

<p>When using an unmodified scale, you can just refer to it. However, if you do make changes, you need to supply a copy of the full modified scale somewhere when you publish. </p>

CC BY-SA 3.0
2014-02-20T23:23:02.273Z
2014-02-21T00:42:15.783Z
2014-02-21T00:42:15.783Z
Jeromy Anglim
<p>I work with questionnaire data and am fairly sensitive to good practice in questionnaire construction. Usually I generate my own, but when I draw on existing literature to use reliably reproduced measurement scales, sometimes the items are poorly worded or vague enough that I would not consider them appropriate or valid. Consider the item:</p>

<p>"Weeds should be eradicated because they inhibit the full development of useful and ornamental plants."</p>

<p>A participant can say that they disagree with this item. So do they think weeds shouldn't be eradicated? Or that they <em>should</em> be eradicated, but not for the reason given. Maybe they think that weeds <em>do</em> inhibit the full development of useful plants, but that is not sufficient cause for eradication of weeds. The question is vague enough that it doesn't confer any useful information about the participant's broader view on weeds except in the one fringe case where they agree with it as stated. Clearly it should be split into two items and already the scale is much improved by it.</p>

<p>There are other examples that are much more shockingly bad, but have been reproduced enough times that they form reliable scales. Is it better to use them as-is, with all their flaws, or to use them as fully as possible while fixing any glaring errors, or not to use them at all?</p>

<measurement>
What is the appropriate action when using poorly worded scales from previous literature?
Adb
# What is the appropriate action when using poorly worded scales from previous literature?

(Asked by: username_0 on 2014-02-20)

I work with questionnaire data and am fairly sensitive to good practice in questionnaire construction. Usually I generate my own, but when I draw on existing literature to use reliably reproduced measurement scales, sometimes the items are poorly worded or vague enough that I would not consider them appropriate or valid. Consider the item:

"Weeds should be eradicated because they inhibit the full development of useful and ornamental plants."

A participant can say that they disagree with this item. So do they think weeds shouldn't be eradicated? Or that they *should* be eradicated, but not for the reason given. Maybe they think that weeds *do* inhibit the full development of useful plants, but that is not sufficient cause for eradication of weeds. The question is vague enough that it doesn't confer any useful information about the participant's broader view on weeds except in the one fringe case where they agree with it as stated. Clearly it should be split into two items and already the scale is much improved by it.

There are other examples that are much more shockingly bad, but have been reproduced enough times that they form reliable scales. Is it better to use them as-is, with all their flaws, or to use them as fully as possible while fixing any glaring errors, or not to use them at all?

**Tags:** measurement


---

## 2 answers

### Answer by username_1 on 2014-02-21. Score: 2

# Intro

Here's a slightly more compromising (yet arguably more demanding) answer than what's, which is probably a good representative of how the idea of changing items will generally be received by reviewers, supervisors, or substantive theorists. From the perspective of a methodology nerd (don't mean to deny @what that dubious distinction too BTW), there are some reasons to consider reworking this questionnaire and certainly very many other measures in common usage. It's true that this changes the latent construct and threatens its comparability to that estimated in previous research, but changes may be justified, and comparability may be worth sacrificing or even entirely undesirable if the measurement model in previous research was just that badly invalid.

It's also true that there is a lot of sophisticated, challenging, expensive work to do in the construct validation process, but as such, much of this work never gets done. For both theoretical and practical reasons, as much of this work as is feasible should be completed, reported, and kept up-to-date, and it's not necessarily any less important to do when one isn't motivated to change the measure. Building on a long line of previous research by using old methods mostly protects comparability of results within that line of research; it doesn't protect the line of research as a whole from its systemic flaws. Enough long lines of heralded theory have fallen over time to discourage complacency based on unquestioning deference to seemingly established methods.

# Scrutiny of [Milfont & Duckitt (2004)](http://gip.uniovi.es/gdiyad/docume/arti_jep01.pdf)

## *The question in question is questionable.*

To demonstrate the methodological vulnerabilities of even relatively modern research, I'll nitpick your specific example. First, as you've noted, the focal question itself is [**double- or even triple-barreled**](https://en.wikipedia.org/wiki/Double-barreled_question):

1. Should weeds be eradicated?
2. Do weeds inhibit the full development of *useful* plants?
   * Do weeds inhibit the full development of *ornamental* plants?

This decreases interpretability of results somewhat needlessly (the measure is long enough already that one extra question wouldn't hurt noticeably...and what good does the distinction between useful and ornamental plants do?) and probably increases measurement error, as respondents are likely to rate their agreement somewhat haphazardly based on whichever part of the question elicits the strongest attitudinal response.

Second, this particular item loads rather weakly ($\lambda=.38)$ on a factor (the tenth and least reliable factor, $\alpha=.58$) with no negatively-weighted items. [Milfont and Duckitt (2004)](http://gip.uniovi.es/gdiyad/docume/arti_jep01.pdf) critique another measure for possessing unbalanced subscales, which by their own admission are "open to [**acquiescence bias**](https://en.wikipedia.org/wiki/Acquiescence_bias)". Yet they haven't fully balanced their own subscales, including also those for factors 2, 4, and 6 – this is just one shy of the number of unbalanced subscales in the measure they critique!

Regardless, "balancing" subscale items doesn't eliminate acquiescence bias; it only "balances" its effects on latent factor scores to the extent that they are calculated based on an equal number of equally-and-oppositely weighted items. In the authors' own usage of the measure (the structural equation model in Figure 1, p. 299), it appears that factor scores were based on the items weighted by their factor loadings, which are not equal. Hence even the factors that do have equal numbers of oppositely-weighted items (of which there are only two: factors 3 and 8) do not truly balance the effects of acquiescence bias.

Furthermore, even the false ideal of a set of items with equal, opposite, and balanced weighting would not truly balance acquiescence effects, as these also depend on item content and respondents' item-specific expertise (McClendon, 1991a), both of which vary across items, of course. In the case of our double/triple-barreled question, the second/third barrel particularly concerns horticultural knowledge, so this item seems likely to suffer specially from acquiescence bias due to that second/third barrel that arguably does more harm than good. Personally, I'm somewhat surprised to find theoretical basis for suspecting that this expertise issue with one (or two) particular barrel(s) causes more acquiescence problems than the ambiguity of the question overall (McClendon, 1991b), as is due to its compound nature. Acquiescence may be measurable and controllable (Billiet & McClendon, 2000), but it's not always a big threat, it's not the only response-style-based threat ([Hinz, Michalski, Schwarz, & Herzberg, 2007](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2736523/)), and balancing oppositely-weighted items halfheartedly is surely an incomplete amelioration (Schriesheim & Hill, 1981).

## *The model is not a model model.*

This leads to further concerns with the overall measurement model. It seems to have been developed with methods that are somewhat **inappropriate for Likert rating data**, which are technically ordinal (i.e., discrete and ranked). The authors don't specify how exactly they handled item scoring, so they probably gave it the simple, conventional treatment as continuous data. This is not altogether indefensible with a seven-point Likert scale (see [this answer on Cross Validated](https://stats.stackexchange.com/a/86959/32036)), but it's not ideal. The particular item in question appears on a scale with only four items, which is probably insufficient for producing a scale score by simple summation or averaging according to [classical test theory (CTT)](http://en.wikipedia.org/wiki/Classical_test_theory); this may further explain the scale's relatively dismal $\alpha$ reliability. Nonetheless, the authors probably used some such CTT method to calculate the scale scores they tested for gender differences in their Table 4, as they haven't referred to these as factor scores, much less specified which kind of factor scores they might be (see [DiStefano, Zhu, & Mîndrilă, (2009)](http://pareonline.net/pdf/v14n20.pdf) for a review of six popular options).

Moreover, the scale scoring system appears to have been developed through application of principal components and principal axis factor extraction from a simple covariance matrix that probably treated the data as continuous, as well as maximum likelihood estimation of a confirmatory factor analytic (CFA) model's fit to that same matrix. See [this question on Cross Validated, "Factor analysis of questionnaires composed of Likert items"](https://stats.stackexchange.com/q/2374/32036) for discussion of preferable methods and other unaddressed issues, including:

* Other estimation algorithms that are less biased by ordinal data than maximum likelihood
* Fitting latent factor models to a polychoric correlation matrix, not a covariance matrix
* Exploration of bifactor structure in the case of a strong general factor amidst multiple dimensions
  + This may be a particularly important alternative to consider alongside a second-order model.
* Preemptive control of various biases, including extreme response style, as via unfolding models

It's also a bit fishy to perform exploratory factor analysis on the same sample as a CFA (see [yet another question on Cross Validated, "Regression testing after dimension reduction"](https://stats.stackexchange.com/q/83512/32036)), though at least Milfont and Duckitt (2004) did this in the less objectionable order to confirm and explore somewhat distinct latent structures. In sum, reasons such as these may explain the authors' failure to achieve good model fit by the conventional standards they cited themselves. When model fit is inadequate, path coefficients may be biased, which may pose several further problems for the proposed latent factor structure.

# General reflection on alternative priorities

In the end, problems such as these vary widely in consequentiality depending on the egregiousness of particular instances. Older measures probably generally suffer more for their age, as many of the methodological issues I've raised here arise from relatively modern research. Unfortunately, this may threaten the validity of retrospective comparisons most where they might otherwise be most richly informative – where lines of research reach back across several decades and even generations – but other sociological factors (e.g., [cohort effects](https://en.wikipedia.org/wiki/Cohort_effect)) may limit the equivalence of old research to new research anyway. Reasons such as these to give up on traditional methods are at least as numerous and important as reasons to adhere loyally.

In more modern instances such as [Milfont and Duckitt (2004)](http://gip.uniovi.es/gdiyad/docume/arti_jep01.pdf), there may be less to lose by parting ways with less established and replicated methods (though I see [their article has been cited at least 180 times already](http://scholar.google.com/scholar?cites=14421768210628985778&as_sdt=2005&sciodt=0,5&hl=en)), but there may also be less to gain. **Some of these issues I've raised may be fairly inconsequential, or if resolved, might even absolve their theory** by improving model fit and demonstrating a lack of cause for further concern. One simply can't be sure without trying.

How much of this effort is worthwhile is an open question, a balancing act, a fundamental career choice, and even a bit of a gamble. Deep, unresolved, and even unresolvable epistemic problems underlie every application of empirical scientific methodology – this may be especially true of the social sciences – but time spent on basic research is not generally time spent directly on application of theory to contemporarily exigent problems. Thus one confronts the **fundamental dilemma of addressing "real-world problems" now vs. theoretical problems for future applications**, and through it the career choice of applied vs. theoretical and methodological research.

One way to strike a balance is to prognosticate the societal impact of your options, and in any given choice to go with the option most likely to matter. If your methodological problem seems more pragmatically consequential than your applied problem, solve it first, or instead if necessary! (This I've stated as an idealized, no-brainer platitude, but intended to reflect a subtler judgment call.) Another is to contribute in the ways you value most – the ways most motivating for you, be they immediate career advancement or long-term legacy – as service to your own values is service you're most likely to perform best and most valuably to all beneficiaries (yourself included). Another is to consider your implicit motives and core strengths: **what do you like to do, and what do you do best?** These factors will affect the value of your work too, and should inform your choices of basic priorities and specific endeavors.

Hopefully you can find ways to make these priorities align in your work, but everyone's bound to face conflicts. I'm working through some of these conflicts presently myself – hence the indulgence in the preceding exegesis – but I'll spare you further autobiography. Ponder instead [the wisdom of Peppy Hare](https://www.youtube.com/watch?v=txWWi2n76FI), and let me know if you've got some data and want to collaborate on updating your measurement models!

**References**

- Billiet, J. B., & McClendon, M. J. (2000). Modeling acquiescence in measurement models for two balanced sets of items. *Structural Equation Modeling, 7*(4), 608–628.  
- DiStefano, C., Zhu, M., & Mîndrilă, D. (2009). Understanding and using factor scores: Considerations for the applied researcher. *Practical Assessment, Research & Evaluation, 14*(20), 1–11. Retrieved from <http://pareonline.net/pdf/v14n20.pdf>.  
- Hinz, A., Michalski, D., Schwarz, R., & Herzberg, P. Y. (2007). The acquiescence effect in responding to a questionnaire. *GMS Psycho-Social Medicine, 4*. Retrieved from <http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2736523/>.  
- McClendon, M. J. (1991a). Acquiescence and recency response-order effects in interview surveys. *Sociological Methods & Research, 20*(1), 60–103.  
- McClendon, M. J. (1991b). Acquiescence: Tests of the cognitive limitations and question ambiguity hypotheses. *Journal of Official Statistics, 7*(2), 153–166.  
- Milfont, T. L., & Duckitt, J. (2004). The structure of environmental attitudes: A first- and second-order confirmatory factor analysis. *Journal of Environmental Psychology, 24*, 289–303. Retrieved from <http://gip.uniovi.es/gdiyad/docume/arti_jep01.pdf>.  
- Schriesheim, C. A., & Hill, K. D. (1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. *Educational and Psychological Measurement, 41*(4), 1101–1114.

> **Comments:**
> **username_0**: This is an excellent writeup. Thank you!  
> **username_1**: Glad it was helpful!  

---

### Answer by username_2 on 2014-02-20. Score: 1

To reiterate @what's answer, here are a few simple rules.

### Change wording when...

* You want to improve the scale and are prepared to engage in a thorough validation process of the new scale. E.g., comparing the new with the old version; re-running factor analyses, reliability analyses, validation. Naturally you'd only do this if you felt that you could make a substantial improvement to the scale.
* You need to substantially adapt a scale to your context. E.g., the test was looking at attitudes to X and you want to look at attitudes to Y. You only have room in your scale for a subset of the items. In all these cases, you are only loosely drawing on the existing validity information for the scale. And adapting a scale to your context should not be your first choice if there is an existing scale already suited to your context.

### Don't change wording when...

* You think that minor changes would make the scale better (e.g., by adding/removing an item; tweaking the wording). Such changes prevent you from comparing your data to other normative studies. While if the changes are minor, the changes to the psychometrics are also likely to be minor, this is ultimately an empirical question. And thus, you are limited in drawing on the psychometric evidence of other studies after you have made changes. Thus, it is rarely worth it.

### Thoughts on changes

When using an unmodified scale, you can just refer to it. However, if you do make changes, you need to supply a copy of the full modified scale somewhere when you publish.


---

