<p>Since this is a relatively new problem for behavioral researchers, I don't know that there is a common consensus.  I found two articles, one of which was a study that had used crowdsourcing for medical pictograms.  </p>

<p>Their approach was as follows:</p>

<blockquote>
  <p>First, we checked for duplicate records. After sorting the data by participants’ IP addresses, we found three pairs of responses with the same IP address. In two pairs, the pictogram interpretations and the demographic survey answers were nearly identical, but the participation dates were different. We counted them as duplicate records and kept only the first record of each on file.</p>
</blockquote>

<p>From:</p>

<blockquote>
  <p>Yu B, Willis M, Sun P, Wang J (2013)
  Crowdsourcing Participatory Evaluation of Medical Pictograms Using Amazon Mechanical Turk
  J Med Internet Res, 15(6):e108   <a href="http://www.jmir.org/2013/6/e108/" rel="nofollow">[FREE]</a> <a href="http://dx.doi.org/10.2196/jmir.2513" rel="nofollow">[DOI]</a></p>
</blockquote>

<p>In an article just posted this month, the subject is dealt with in more general terms, and some statistics are offered as to how frequently this might occur, and some of the reasons behind it.</p>

<blockquote>
  <p>Although workers may be able to have more than one concurrent MTurk account and, thus, more than one WorkerID,
  this is uncommon. Amazon actively works to identify and
  eliminate duplicate accounts. More important, requesters
  often restrict lucrative HITs to workers who have completed
  a large volume of high-quality work in the past</p>
</blockquote>

<p>So, it seems that WorkerID can be used as a unique identifier and Amazon does actually screen for duplicate accounts, eliminating some of the risk of duplicate responders to the same study.</p>

<p>In terms of IP addresses, the article offers some idea of the extent of the problem:</p>

<blockquote>
  <p>Examinations of worker IP addresses typically reveal a small minority of
  workers (around 2.5 %; Berinsky et al.,2012**) who submit HITs from the same IP address, which may often result from workers being separate members of a single household. </p>
</blockquote>

<p>From:</p>

<blockquote>
  <p>Chandler, J, Mueller, P, Paolacci, G (2013). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers.  Behavior Research Methods, published online 9 July 2013 <a href="http://dx.doi.org/10.3758/s13428-013-0365-7" rel="nofollow">[DOI]</a></p>
</blockquote>

<p>The first article pointed out the fact that dropping data from a study based upon duplicate IP addresses can be done without much of a cost penalty.  </p>

<p>Other demographic factors can be used to discern whether these are different people, but the accuracy of these answers is not guaranteed, but can be bolstered by having dependencies between demographic categories (e.g., recording gender and last menstrual period should be consistent).</p>

<p>Results should be checked more carefully for users that share the same IP address, as even if they are different members of a household, they may be sharing a set of answers, but it seems like there is a low cost to including data from those with matching IP addresses.</p>

<p><sup> ** The citation for the Berinsky paper included in the Chandler 2013 work is as follows: Berinsky, A. J., Huber, G. A., &amp; Lenz, G. S. (2012). Evaluating online
labor markets for experimental research: Amazon.com’s
Mechanical Turk. Political Analysis, 20(3), 351–368. <a href="http://dx.doi.org/10.1093/pan/mpr057" rel="nofollow">[DOI]</a>  I did not examine this work</sup> </p>

CC BY-SA 3.0
2013-07-27T09:24:09.463Z
2013-07-27T09:34:25.287Z
2013-07-27T09:34:25.287Z
Chuck Sherrington
<p><em>I saw <a href="https://twitter.com/jayvanbavel/status/360873893843582977" rel="nofollow">this question</a> on Twitter from Jay Van Bavel (@jayvanbavel), and thought it was a good fit for the site.</em></p>

<blockquote>
  <p>Does anyone find that multiple subjects from the same IP address in
  their #MTurk data? How do you deal with this issue? Citations welcome.</p>
</blockquote>

<p>Thus, to rephrase:</p>

<ul>
<li>What does it mean when you get two or more Mechanical Turk participants in a study with the same IP address?</li>
<li>What is a general strategy for dealing with such cases?</li>
<li>Are there are any references discussing the issue?</li>
</ul>

CC BY-SA 3.0
2013-07-27T09:17:45.287Z
This question is wildly off-topic
awdz9nld
CC BY-SA 3.0
2013-07-27T09:24:51.713Z
@MartinKällman Why do you say so, this is definitely a critical part of modern research methodology in the cognitive sciences.
Chuck Sherrington
CC BY-SA 3.0
2013-07-28T06:14:45.923Z
See http://stackoverflow.com/questions/2835915/in-mechanical-turk-how-do-you-limit-to-one-hit-per-worker
John Pick
<methodology><internet><mechanical-turk>
How should multiple Mechanical Turk research subjects from the same IP address be handled?
Jeromy Anglim
# How should multiple Mechanical Turk research subjects from the same IP address be handled?

(Asked by: username_0 on 2013-07-27)

*I saw [this question](https://twitter.com/jayvanbavel/status/360873893843582977) on Twitter from Jay Van Bavel (@jayvanbavel), and thought it was a good fit for the site.*

> Does anyone find that multiple subjects from the same IP address in
> their #MTurk data? How do you deal with this issue? Citations welcome.

Thus, to rephrase:

* What does it mean when you get two or more Mechanical Turk participants in a study with the same IP address?
* What is a general strategy for dealing with such cases?
* Are there are any references discussing the issue?

**Tags:** methodology, internet, mechanical-turk

> **Comments:**
> **username_1**: This question is wildly off-topic  
> **username_2**: @MartinKällman Why do you say so, this is definitely a critical part of modern research methodology in the cognitive sciences.  
> **username_3**: See http://stackoverflow.com/questions/2835915/in-mechanical-turk-how-do-you-limit-to-one-hit-per-worker  

---

## 1 answers

### Answer by username_2 on 2013-07-27. Score: 4 | Accepted answer

Since this is a relatively new problem for behavioral researchers, I don't know that there is a common consensus. I found two articles, one of which was a study that had used crowdsourcing for medical pictograms.

Their approach was as follows:

> First, we checked for duplicate records. After sorting the data by participants’ IP addresses, we found three pairs of responses with the same IP address. In two pairs, the pictogram interpretations and the demographic survey answers were nearly identical, but the participation dates were different. We counted them as duplicate records and kept only the first record of each on file.

From:

> Yu B, Willis M, Sun P, Wang J (2013)
> Crowdsourcing Participatory Evaluation of Medical Pictograms Using Amazon Mechanical Turk
> J Med Internet Res, 15(6):e108 [[FREE]](http://www.jmir.org/2013/6/e108/) [[DOI]](http://dx.doi.org/10.2196/jmir.2513)

In an article just posted this month, the subject is dealt with in more general terms, and some statistics are offered as to how frequently this might occur, and some of the reasons behind it.

> Although workers may be able to have more than one concurrent MTurk account and, thus, more than one WorkerID,
> this is uncommon. Amazon actively works to identify and
> eliminate duplicate accounts. More important, requesters
> often restrict lucrative HITs to workers who have completed
> a large volume of high-quality work in the past

So, it seems that WorkerID can be used as a unique identifier and Amazon does actually screen for duplicate accounts, eliminating some of the risk of duplicate responders to the same study.

In terms of IP addresses, the article offers some idea of the extent of the problem:

> Examinations of worker IP addresses typically reveal a small minority of
> workers (around 2.5 %; Berinsky et al.,2012\*\*) who submit HITs from the same IP address, which may often result from workers being separate members of a single household.

From:

> Chandler, J, Mueller, P, Paolacci, G (2013). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, published online 9 July 2013 [[DOI]](http://dx.doi.org/10.3758/s13428-013-0365-7)

The first article pointed out the fact that dropping data from a study based upon duplicate IP addresses can be done without much of a cost penalty.

Other demographic factors can be used to discern whether these are different people, but the accuracy of these answers is not guaranteed, but can be bolstered by having dependencies between demographic categories (e.g., recording gender and last menstrual period should be consistent).

Results should be checked more carefully for users that share the same IP address, as even if they are different members of a household, they may be sharing a set of answers, but it seems like there is a low cost to including data from those with matching IP addresses.

\*\* The citation for the Berinsky paper included in the Chandler 2013 work is as follows: Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online
labor markets for experimental research: Amazon.com’s
Mechanical Turk. Political Analysis, 20(3), 351–368. [[DOI]](http://dx.doi.org/10.1093/pan/mpr057) I did not examine this work


---

