<p><strong>Preventing random responding:</strong> An important first step is to think about ways to prevent random responding from occurring in the first place. A few ideas include: administer the survey face to face; have an experimental invigilator present; communicate the importance of the research to participants and the importance of participants taking the research seriously; use financial remuneration.</p>

<p>That said, there are situations where participants do not take a study seriously responding randomly for example. This seems to be particularly an issue when collecting data online.</p>

<p><strong>General approach</strong>: My overall approach to this is to develop multiple indicators of problematic participation. I'll then assign penalty points to each participant based on the severity of the indicators. Participants with penalty points above a threshold are excluded from analyses.</p>

<p>The choices of what is problematic depends on the type of study:</p>

<ul>
<li>If a study is performed in a <strong>face to face</strong> setting, the experimenter can take notes recording when participants engage in problematic behaviour.</li>
<li>In <strong>online survey</strong> style studies I record reaction time for each item. I then see how many items are answered more quickly than the person could conceivably read and respond to the item. For example, answering a personality test item in less than about 600 or even 800 milliseconds indicates that the participant has skipped an item. I then count up the number of times this occurs, and set a cut-off.</li>
<li>In <strong>performance based tasks</strong>, other participant actions may imply distraction or not taking the task seriously. I'll try to develop indicators for this.</li>
</ul>

<p><a href="http://en.wikipedia.org/wiki/Mahalanobis_distance" rel="nofollow">Mahalanobis distance</a> is often a useful tool to flag multivariate outliers. You can further inspect the cases with the largest values to think about whether they make sense. There is a bit of an art in deciding which variables to include in the distance calculation. In particular, if you have a mix of positively and negatively worded items, carelessness is often indicated by a lack movement between the poles of a scale as you move from positively to negatively worded items.</p>

<p>In general, I also often include items at the end of the test asking the participant whether they took the experiment seriously.</p>

<h3>Discussion in the Literature</h3>

<p>Osborne and Blanchard (2010) discuss random responding in the context of multiple choice tests. They mention the strategy of including items that all participants should answer correctly. To quote</p>

<blockquote>
  <p>These can be content that should not be missed (e.g., 2+2=__),
  behavioral/attitudinal questions (e.g., I weave the fabric for all my
  clothes), non-sense items (e.g., there are 30days in February), or
  targeted multiple-choice test items [e.g., “How do you spell
  ‘forensics’?” (a) fornsis, (b) forensics, (c) phorensicks, (d)
  forensix].</p>
</blockquote>

<h3>References</h3>

<ul>
<li>Osborne, J. W., &amp; Blanchard, M. R. (2010). Random responding from participants is a threat to the validity of social science research results. Frontiers in Psychology, 1. <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3153825/" rel="nofollow">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3153825/</a></li>
</ul>

CC BY-SA 3.0
2013-03-10T08:51:27.377Z
For surveys, how do you use an "invigilator" or "take notes recording when participants engage in problematic behaviour," without violating the anonymity of the participant?
Ryan Lang
CC BY-SA 3.0
2013-03-10T07:03:05.843Z
2013-05-07T12:21:12.103Z
2013-05-07T12:21:12.103Z
Jeromy Anglim
<p>This is not directly an answer to your question but, in line with my comments to another answer, my main advice would be “don't worry about it”.</p>

<p>Jeromy Anglim's tips are all good but I am still unconvinced that this is an important issue for most people. Since you are new to research, there are probably dozens of other things you should worry about.</p>

<p>Furthermore, if you do see evidence that there is a problem (extremely short response times, contradictory answers, large number of respondent providing absurd answers to open-ended questions), I would argue that you should first step back and ask yourself if what you are asking is reasonable (Do the task make sense? Can people be expected to have an opinion about the topic you are investigating? Are you demanding too much effort?) rather than trying to sort out “bad” respondents.</p>

<p>If you really want to dig into the subject and look up some literature, another name for this phenomenon is “satisficing”. “Response set” is a related idea that might be of interest.</p>

CC BY-SA 3.0
2013-05-11T21:29:32.453Z
agreed, this should definitely be a first step before "correcting" the "problem"
Jeff
CC BY-SA 3.0
2013-05-07T18:56:05.983Z
2013-05-07T18:56:05.983Z
Gala
<p>You seem to be concerned with <a href="http://en.wikipedia.org/wiki/Reliability_(psychometrics)" rel="nofollow noreferrer">reliability</a>, and more specifically <a href="http://en.wikipedia.org/wiki/Internal_consistency" rel="nofollow noreferrer">internal reliability</a>. Internal reliability is the degree to which different questions are measuring the same construct. This concept is used often in psychology and is usually measured using <a href="http://en.wikipedia.org/wiki/Cronbach%27s_alpha" rel="nofollow noreferrer">Cronbach's alpha</a>. However, it is typically used to measure the reliability of a <em>test</em>, and not the reliability of an <em>individual</em>.</p>

<p>As Jeromy Anglim points out, I think it's important to consider the goal here. Using a two question Likert scale is probably not good enough to reliably detect outliers: What if the respondent checked all '4s' on a 7-point Likert scale? Reversing the scale would have no effect.</p>

<p>One alternative approach is to employ an <strong>instructional manipulation check</strong> (Oppenheimer et al., 2009). The gist of the technique is to trap participants into answering a question in a <em>specific way</em> that they could only have done by reading the instructions carefully. Here is an example from a survey administered by Facebook:</p>

<p><img src="https://i.sstatic.net/9qFm0.png" alt="enter image description here"></p>

<p>While this technique may throw out a few good participants, it will almost certainly raise the signal-to-noise ratio of your data by only including participants who followed instructions and read questions before answering.</p>

<p>Another tried and true technique is to use a computer-administered test and look at reaction times. You may be able to throw out a few responses (or whole participants) by simply looking for outliers in response time that are below the mean.</p>

<blockquote>
  <p>Oppenheimer, D. M., Meyvis, T., &amp; Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867-872.</p>
</blockquote>

CC BY-SA 3.0
2013-03-11T05:14:04.780Z
"Internal reliability tests the degree to which different questions are measuring the same construct" does not seem quite right. You can have 2 underlying dimensions and have high Cronbach's alpha. See: http://psycnet.apa.org/journals/pas/8/4/350/
RJ-
CC BY-SA 3.0
2013-03-11T05:47:54.633Z
@RJ that means Cronbach's alpha might not be measuring internal reliability, not that the definition of internal reliability is wrong. According to the paper you cite, "Internal consistency refers to the interrelatedness of a set of items" which seems in line with what I am saying.
Jeff
CC BY-SA 3.0
2013-03-11T05:50:36.227Z
I am taking issue mainly with "measuring the same construct". The paper also points out that "measuring the same construct" is different from "inter-relatedness" which is what Cronbach's alpha is measuring.
RJ-
CC BY-SA 3.0
2013-03-11T05:58:25.043Z
Ah, perhaps I should change the wording to "Internal reliability **is** the degree to which..." and "Cronbach's alpha **tests**..." I can see how my definition is not in line with what Cronbach's alpha is *testing*, but still think it's an accurate description of what internal reliability *is*.
Jeff
CC BY-SA 3.0
2013-03-11T19:36:34.797Z
@what that is sort of the point. many psychology experiments contain blocks of text that explain how to answer the following questions. it's important that you read it to answer them correctly. if you skip sections that you don't "expect to have relevant instructions" then, well, you may be skipping relevant instructions. case in point, the facebook survey. however, this is not the only example of an IMC; try also reading the oppenheimer reference or do a scholar search for "instructional manipulation check"
Jeff
CC BY-SA 3.0
2013-03-11T21:35:28.780Z
I agree that good UI is important. There are many reasons why a participant may not follow instructions, and bad UI is one of them. But from the data analyst's standpoint, it really doesn't matter what the reason is. The IMC detects failure to follow instructions for any reason.
Jeff
CC BY-SA 3.0
2013-03-11T21:39:52.303Z
As an aside, in many experiments, instructions will apply to a block of questions. Putting instructions in between a question and its answers is not feasible. I do agree that the heading title itself "Almost  done!" is filler, which probably leads a lot of people to skip that section. It should probably say "Instructions". But I don't think the placement of the any of the elements is a problem here. In any case, this is a debate for [chat](http://chat.stackexchange.com/rooms/2250/the-axon-terminal) or for ux.SE and doesn't really have a bearing on my answer IMHO.
Jeff
CC BY-SA 3.0
2013-05-07T18:33:44.340Z
(-1) I think this is terrible advice. The question is only tangentially related to reliability. Cronbach alpha, beside being much less useful than usually thought and often misinterpreted, does not address it at all. Alpha, internal consistency or reliability all come up when building or interpreting a scale and can only be computed over a set of scores. None of this helps to select *observations*.
Gala
CC BY-SA 3.0
2013-05-07T18:37:32.953Z
The only relevant part is the Facebook trap question but it looks like the cure is worse than the disease here. You risk confusing a great number of sincere respondents (and you certainly can't assume that the “good” respondents you exclude are picked at random so that you are not only reducing power and sample size but also introducing bias) for a benefit that is very doubtful.
Gala
CC BY-SA 3.0
2013-05-07T18:43:16.793Z
Researchers are prone to worry about it but I am yet to see evidence that satisficing is generally such a big problem (I don't see it in my research and I have run all sorts of psychological experiments with students, usability tests, long interviews with people of all ages, crowdsourcing studies on Mechanical Turk, Internet surveys in more than a dozen countries, etc.)
Gala
CC BY-SA 3.0
2013-05-07T19:53:17.970Z
@GaëlLaurans I am *not* suggesting the use of Cronbach's alpha here. I mentioned it because it seems like that's the solution OP was *trying* to employ, but as I said in my post it is inappropriate here. As for the Facebook example-- don't take it too literally. I'm promoting the general idea behind an IMC, and not advocating for a specific implementation. There are probably more effective examples.
Jeff
CC BY-SA 3.0
2013-05-07T19:54:24.290Z
As for the issue of satisficing, I'm simply answering the OP's question. He is concerned about it, and other researchers have voiced concerns as well. I never claimed that it was a widespread problem, but it appears to be a problem in at least some circumstances.
Jeff
CC BY-SA 3.0
2013-03-11T02:33:16.147Z
2013-03-11T10:04:14.233Z
2013-03-11T10:04:14.233Z
Jeff
<p>I have an experimental study with a list of demographic and related questions and in order to identify data from participants that were potentially just answering the questions at random (to get through them more quickly I would assume), I've included two very similar 7-point likert scale questions at different points in the survey. My assumption would be that since the questions are reflective, the answers participants should give will be at least somewhat similar between the two questions (eg, it should be very unlikely that a participant answers 7 to one question yet 1 to the other).</p>

<p>I haven't yet collected the data, however I would like to have a method for determining which sets of data are suspicious (might be considered for exclusion in analysis) based on these control questions. One method might be to simply determine where the data fit on a Gaussian distribution. However, I think that the limited discriminating power of a 7-point scale would make this an improper test. My other idea was to do a cluster analysis on the data, looking for five groups: three along the line of correlation (between the questions), and two to examine unusually high/low and low/high values. I thought this could provide better suggestions for which data sets might be unusual since it wouldn't use somewhat arbitrary comparisons, it would only use the data given.</p>

<p>I'd really appreciate any suggestions for a better method, or improvements I could make as well as any comments toward more "standard" practices in this area, since I'm somewhat new to research. </p>

<methodology><survey>
Analyzing control questions data for a survey
Ryan Lang
# Analyzing control questions data for a survey

(Asked by: username_0 on 2013-03-09)

I have an experimental study with a list of demographic and related questions and in order to identify data from participants that were potentially just answering the questions at random (to get through them more quickly I would assume), I've included two very similar 7-point likert scale questions at different points in the survey. My assumption would be that since the questions are reflective, the answers participants should give will be at least somewhat similar between the two questions (eg, it should be very unlikely that a participant answers 7 to one question yet 1 to the other).

I haven't yet collected the data, however I would like to have a method for determining which sets of data are suspicious (might be considered for exclusion in analysis) based on these control questions. One method might be to simply determine where the data fit on a Gaussian distribution. However, I think that the limited discriminating power of a 7-point scale would make this an improper test. My other idea was to do a cluster analysis on the data, looking for five groups: three along the line of correlation (between the questions), and two to examine unusually high/low and low/high values. I thought this could provide better suggestions for which data sets might be unusual since it wouldn't use somewhat arbitrary comparisons, it would only use the data given.

I'd really appreciate any suggestions for a better method, or improvements I could make as well as any comments toward more "standard" practices in this area, since I'm somewhat new to research.

**Tags:** methodology, survey


---

## 3 answers

### Answer by username_3 on 2013-03-11. Score: 4 | Accepted answer

You seem to be concerned with [reliability](http://en.wikipedia.org/wiki/Reliability_(psychometrics)), and more specifically [internal reliability](http://en.wikipedia.org/wiki/Internal_consistency). Internal reliability is the degree to which different questions are measuring the same construct. This concept is used often in psychology and is usually measured using [Cronbach's alpha](http://en.wikipedia.org/wiki/Cronbach%27s_alpha). However, it is typically used to measure the reliability of a *test*, and not the reliability of an *individual*.

As username_1 points out, I think it's important to consider the goal here. Using a two question Likert scale is probably not good enough to reliably detect outliers: What if the respondent checked all '4s' on a 7-point Likert scale? Reversing the scale would have no effect.

One alternative approach is to employ an **instructional manipulation check** (Oppenheimer et al., 2009). The gist of the technique is to trap participants into answering a question in a *specific way* that they could only have done by reading the instructions carefully. Here is an example from a survey administered by Facebook:

![enter image description here](https://i.sstatic.net/9qFm0.png)

While this technique may throw out a few good participants, it will almost certainly raise the signal-to-noise ratio of your data by only including participants who followed instructions and read questions before answering.

Another tried and true technique is to use a computer-administered test and look at reaction times. You may be able to throw out a few responses (or whole participants) by simply looking for outliers in response time that are below the mean.

> Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867-872.

> **Comments:**
> **username_4**: "Internal reliability tests the degree to which different questions are measuring the same construct" does not seem quite right. You can have 2 underlying dimensions and have high Cronbach's alpha. See: http://psycnet.apa.org/journals/pas/8/4/350/  
> **username_3**: @RJ that means Cronbach's alpha might not be measuring internal reliability, not that the definition of internal reliability is wrong. According to the paper you cite, "Internal consistency refers to the interrelatedness of a set of items" which seems in line with what I am saying.  
> **username_4**: I am taking issue mainly with "measuring the same construct". The paper also points out that "measuring the same construct" is different from "inter-relatedness" which is what Cronbach's alpha is measuring.  
> **username_3**: Ah, perhaps I should change the wording to "Internal reliability **is** the degree to which..." and "Cronbach's alpha **tests**..." I can see how my definition is not in line with what Cronbach's alpha is *testing*, but still think it's an accurate description of what internal reliability *is*.  
> **username_3**: @what that is sort of the point. many psychology experiments contain blocks of text that explain how to answer the following questions. it's important that you read it to answer them correctly. if you skip sections that you don't "expect to have relevant instructions" then, well, you may be skipping relevant instructions. case in point, the facebook survey. however, this is not the only example of an IMC; try also reading the oppenheimer reference or do a scholar search for "instructional manipulation check"  
> **username_3**: I agree that good UI is important. There are many reasons why a participant may not follow instructions, and bad UI is one of them. But from the data analyst's standpoint, it really doesn't matter what the reason is. The IMC detects failure to follow instructions for any reason.  
> **username_3**: As an aside, in many experiments, instructions will apply to a block of questions. Putting instructions in between a question and its answers is not feasible. I do agree that the heading title itself "Almost  done!" is filler, which probably leads a lot of people to skip that section. It should probably say "Instructions". But I don't think the placement of the any of the elements is a problem here. In any case, this is a debate for [chat](http://chat.stackexchange.com/rooms/2250/the-axon-terminal) or for ux.SE and doesn't really have a bearing on my answer IMHO.  
> **username_2**: (-1) I think this is terrible advice. The question is only tangentially related to reliability. Cronbach alpha, beside being much less useful than usually thought and often misinterpreted, does not address it at all. Alpha, internal consistency or reliability all come up when building or interpreting a scale and can only be computed over a set of scores. None of this helps to select *observations*.  
> **username_2**: The only relevant part is the Facebook trap question but it looks like the cure is worse than the disease here. You risk confusing a great number of sincere respondents (and you certainly can't assume that the “good” respondents you exclude are picked at random so that you are not only reducing power and sample size but also introducing bias) for a benefit that is very doubtful.  
> **username_2**: Researchers are prone to worry about it but I am yet to see evidence that satisficing is generally such a big problem (I don't see it in my research and I have run all sorts of psychological experiments with students, usability tests, long interviews with people of all ages, crowdsourcing studies on Mechanical Turk, Internet surveys in more than a dozen countries, etc.)  
> **username_3**: @GaëlLaurans I am *not* suggesting the use of Cronbach's alpha here. I mentioned it because it seems like that's the solution OP was *trying* to employ, but as I said in my post it is inappropriate here. As for the Facebook example-- don't take it too literally. I'm promoting the general idea behind an IMC, and not advocating for a specific implementation. There are probably more effective examples.  
> **username_3**: As for the issue of satisficing, I'm simply answering the OP's question. He is concerned about it, and other researchers have voiced concerns as well. I never claimed that it was a widespread problem, but it appears to be a problem in at least some circumstances.  

---

### Answer by username_1 on 2013-03-10. Score: 3

**Preventing random responding:** An important first step is to think about ways to prevent random responding from occurring in the first place. A few ideas include: administer the survey face to face; have an experimental invigilator present; communicate the importance of the research to participants and the importance of participants taking the research seriously; use financial remuneration.

That said, there are situations where participants do not take a study seriously responding randomly for example. This seems to be particularly an issue when collecting data online.

**General approach**: My overall approach to this is to develop multiple indicators of problematic participation. I'll then assign penalty points to each participant based on the severity of the indicators. Participants with penalty points above a threshold are excluded from analyses.

The choices of what is problematic depends on the type of study:

* If a study is performed in a **face to face** setting, the experimenter can take notes recording when participants engage in problematic behaviour.
* In **online survey** style studies I record reaction time for each item. I then see how many items are answered more quickly than the person could conceivably read and respond to the item. For example, answering a personality test item in less than about 600 or even 800 milliseconds indicates that the participant has skipped an item. I then count up the number of times this occurs, and set a cut-off.
* In **performance based tasks**, other participant actions may imply distraction or not taking the task seriously. I'll try to develop indicators for this.

[Mahalanobis distance](http://en.wikipedia.org/wiki/Mahalanobis_distance) is often a useful tool to flag multivariate outliers. You can further inspect the cases with the largest values to think about whether they make sense. There is a bit of an art in deciding which variables to include in the distance calculation. In particular, if you have a mix of positively and negatively worded items, carelessness is often indicated by a lack movement between the poles of a scale as you move from positively to negatively worded items.

In general, I also often include items at the end of the test asking the participant whether they took the experiment seriously.

### Discussion in the Literature

Osborne and Blanchard (2010) discuss random responding in the context of multiple choice tests. They mention the strategy of including items that all participants should answer correctly. To quote

> These can be content that should not be missed (e.g., 2+2=\_\_),
> behavioral/attitudinal questions (e.g., I weave the fabric for all my
> clothes), non-sense items (e.g., there are 30days in February), or
> targeted multiple-choice test items [e.g., “How do you spell
> ‘forensics’?” (a) fornsis, (b) forensics, (c) phorensicks, (d)
> forensix].

### References

* Osborne, J. W., & Blanchard, M. R. (2010). Random responding from participants is a threat to the validity of social science research results. Frontiers in Psychology, 1. <http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3153825/>

> **Comments:**
> **username_0**: For surveys, how do you use an "invigilator" or "take notes recording when participants engage in problematic behaviour," without violating the anonymity of the participant?  

---

### Answer by username_2 on 2013-05-07. Score: 2

This is not directly an answer to your question but, in line with my comments to another answer, my main advice would be “don't worry about it”.

username_1's tips are all good but I am still unconvinced that this is an important issue for most people. Since you are new to research, there are probably dozens of other things you should worry about.

Furthermore, if you do see evidence that there is a problem (extremely short response times, contradictory answers, large number of respondent providing absurd answers to open-ended questions), I would argue that you should first step back and ask yourself if what you are asking is reasonable (Do the task make sense? Can people be expected to have an opinion about the topic you are investigating? Are you demanding too much effort?) rather than trying to sort out “bad” respondents.

If you really want to dig into the subject and look up some literature, another name for this phenomenon is “satisficing”. “Response set” is a related idea that might be of interest.

> **Comments:**
> **username_3**: agreed, this should definitely be a first step before "correcting" the "problem"  

---

