<p>There are a few factors that could contribute to differences between online versus in-lab reaction time measurement.</p>

<p><strong>Hardware variation</strong></p>

<p>Participants in an online experiment will use their own computers to complete the task, which will result in lots of variation in hardware. Many studies have looked at how hardware variations affect response time measurement, and generally find that hardware variations can cause differences in the range of 10-100ms for a single response (e.g. <a href="http://www.ncbi.nlm.nih.gov/pubmed/19587169" rel="nofollow">Plant &amp; Turner, 2009</a>).</p>

<p><strong>Software variation</strong></p>

<p>Online studies and lab studies tend to be conducted with different software, since most of the standard lab-based software can't be used to make an online experiment. A popular option for online experiments is JavaScript &amp; HTML. <a href="http://www.ncbi.nlm.nih.gov/pubmed/24903687" rel="nofollow">Reimers &amp; Stewart (2014)</a> measured the error in JavaScript response time measurements, and generally found it to be around 25ms, with some variation across different hardware and software. <a href="http://link.springer.com/article/10.3758/s13428-015-0567-2" rel="nofollow">de Leeuw &amp; Motz (2015)</a> ran an experiment in which subjects completed a visual search task in the lab using both a JavaScript and MATLAB (Psychophysics Toolbox) version of the experiment, and found that JavaScript measured response times that were about 25ms slower. However, both JavaScript and MATLAB had equivalent variance in the measurements and both software systems were equally sensitive to experimental manipulations of the visual search task at the sample sizes used for the experiment.</p>

<p><strong>Online versus in the lab</strong></p>

<p><a href="http://dx.doi.org/10.3758/s13428-015-0678-9" rel="nofollow">Hilbig (in press)</a> randomly assigned participants to complete an experiment in the lab using standard lab software (E-prime), in the lab using a web browser, or online at a location of the participant's choice. They measured response times in a standard lexical decision task, and found that there were no significant differences among the three groups. The effect was on the order of 120-220ms. The effect is relatively large (d' ~ 1.5), but given the current literature there is no reason to doubt that it would.</p>

<p><strong>Does it matter?</strong></p>

<p>The last part of the answer is: does it really matter if response times collected online are noisier than those collected in the lab? It turns out that even for pretty noisy measurements, moderate sample sizes will counteract the additional noise of the measurement. Reimers &amp; Stewart (2014) simulated the sample size necessary to find an effect of 50ms with and without the additional noise in response times caused by using online methods. They found that only about 10% more subjects were needed in their model to have an equivalent likelihood of detecting the effect. <a href="http://onlinelibrary.wiley.com/doi/10.1111/j.2044-8317.1989.tb01111.x/full" rel="nofollow">Ulrich &amp; Giray (1989)</a> came to a similar conclusion in a different modeling context.</p>

<p><strong>References</strong></p>

<ul>
<li>de Leeuw, J. R., &amp; Motz, B. A. (2015). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. <em>Behavior Research Methods</em>. <a href="http://dx.doi.org/10.3758/s13428-015-0567-2" rel="nofollow">doi:10.3758/s13428-015-0567-2</a></li>
<li>Hilbig, B. E. (in press). Reaction time effects in lab- versus web-based research: Experimental evidence. <em>Behavior Research Methods</em>. <a href="http://dx.doi.org/10.3758/s13428-015-0678-9" rel="nofollow">doi:10.3758/s13428-015-0678-9</a></li>
<li>Plant, R., &amp; Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: new hardware, new problems? <em>Behavior Research Methods</em>, <em>41</em>(3), 598-614.</li>
<li>Reimers, S., &amp; Stewart, N. (2014). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. <em>Behavior Research Methods</em></li>
<li>Ulrich, R., &amp; Giray, M. (1989). Time resolution of clocks: Effects on reaction time measurement - Good news for bad clocks. <em>British Journal of Mathematical and Statistical Psychology</em>, <em>42</em>, 1-12.</li>
</ul>

CC BY-SA 3.0
2015-03-22T19:54:16.323Z
I'm currently out of votes to up this answer, but I wanted to give a special thanks for sharing your summary of the unpublished data with just the right details (which I will absolutely treat with utmost suspicion until it has been reviewed by three arbitrary persons and locked securely behind an Elsevier paywall). I look forward to reading your paper!
Christian Hummeluhr
CC BY-SA 3.0
2015-03-22T19:29:55.750Z
2016-06-22T13:51:21.340Z
2016-06-22T13:51:21.340Z
Josh
<p>We cover a discussion on this in an article we've submitted for peer-review. Here is the <a href="https://peerj.com/preprints/921/" rel="nofollow noreferrer">preprint</a>.</p>

<p>I will cite this stackExchange question/answers in the manuscript (post peer review now) as there are some lovely discussions going on, and doubtless, more to follow. </p>

<p>Tangentially relevant to this discussion is a simulation we did in the paper exploring how not knowing screen refresh impacts on stimulus timing (consider that if RT <em>starts</em> being recorded from a stimulus presentation, error in stimulus timing mucks around with RT):</p>

<blockquote>
  <p>We tested this appearance issue in a simulation where we varied the
  duration of visual stimulus, starting at a random time during the
  refresh cycle (10,000 virtual presentations per stimulus duration).
  Figure 5 shows the likelihood of short duration stimuli being shown at
  all, or being shown for the wrong duration, or starting / stopping at
  the wrong time (<a href="https://github.com/andytwoods/refreshSimulation" rel="nofollow noreferrer">https://github.com/andytwoods/refreshSimulation</a>;
  available to run / tweak online here
  <a href="http://jsfiddle.net/andytwoods/0f56hmaf/" rel="nofollow noreferrer">http://jsfiddle.net/andytwoods/0f56hmaf/</a>).
  <img src="https://i.sstatic.net/2RctD.png" alt="enter image description here"></p>
</blockquote>

<p>Below is the abstract: </p>

<blockquote>
  <p>This article provides an overview of the literature on the use of
  internet-based testing to address questions in perception research.
  Internet-based testing has several advantages over in-lab research,
  including the ability to reach a relatively broad set of participants
  and to quickly and inexpensively collect large amounts of empirical
  data. In many cases, the quality of online data appears to match that
  collected in laboratory research. Generally speaking, online
  participants tend to be more representative of the population at large
  than laboratory based participants. There are, though, some important
  caveats, when it comes to collecting data online. It is obviously much
  more difficult to control the exact parameters of stimulus
  presentation (such as display characteristics) in online research.
  There are also some thorny ethical considerations that need to be
  considered by experimenters. Strengths and weaknesses of the online
  approach, relative to others, are highlighted, and recommendations
  made for those researchers who might be thinking about conducting
  their own studies using this increasingly-popular approach to research
  in the psychological sciences.</p>
</blockquote>

CC BY-SA 3.0
2015-05-05T10:56:10.560Z
Welcome to CogSci Andy! That would great! To improve this answer, would you mind adding a short synopsis of your recent publication?
Steven Jeuris
CC BY-SA 3.0
2015-05-05T11:10:23.480Z
I  can't tell you how happy I am that you found useful information for your review here (and about you submitting it for preprint). With all the power problems we have in the field, knowing when and how much we can trust online studies will make a huge difference. Based on a skim, it looks like I'll be reading a lot of the papers you cite.
Christian Hummeluhr
CC BY-SA 3.0
2015-05-05T16:57:42.870Z
hope that's sufficient.
andyw
CC BY-SA 3.0
2015-05-05T10:32:17.953Z
2015-05-05T16:49:55.920Z
2015-05-05T16:49:55.920Z
andyw
<p>Depending on how you collect the data, reaction times collected "online" will likely be different from those collected "on-site". When considering reaction times, it is important to decide if the reaction time is being used as a trigger, as the time to a response, or the difference in the time to response.</p>

<p>Consider an experiment which displays a random series of images for a 1/2 second each and the analysis consists of averaging the images that resulted in key presses. If your online system introduces a 1 s delay, you will not be averaging the images that led to the keypress, but the random image after it.</p>

<p>Consider an experiment which displays a random series of images for a 1/2 second each and every once in a while a target image is displayed and the analysis focuses on average amount of time it takes to react to the target image. In this case your reaction time will be 1 second longer than it should be and your data will be effectively meaningless.</p>

<p>Consider an experiment which displays a random series of images for a 1/2 second each and every once in a while one of two target images is displayed and the analysis focuses on difference in the average amount of time it takes to react to the target image. In this case the reaction time to each target will be 1 second longer than it should be, but the difference in reaction will be accurate. If in this experiment, in addition to the 1 s delay, there is also a variable delay (e.g., a Gaussian jitter with mean 0 and variance 1 s). This jitter will add noise to the data and make seeing small difference difficult. This noise, however, will average out across the trials and participants.</p>

<p>As with most psychological measures, the experimenter can trade fidelity of the measurement, number of measurements on each participant, and number of participants against each other. Online studies give up fidelity, and to an extent number of measurements on each participant, for large numbers of participants.</p>

<p>In a lab with a dedicated button box on fast dedicated hardware response latencies of under 1 ms are possible with tiny jitters. This of course ignores the human subject making the response. <a href="http://www.sciencedirect.com/science/article/pii/S0022249605000106" rel="nofollow">Wagenmakers et al (2005)</a> suggest jitter of about 100 ms in the best case. Adding keyboard and network jitter of 100 ms (which would be pretty bad) means the online experiment would need 2x increase in the number of subjects to have the same statistical power; if the variance is twice as big (which happens when you add two independent sources of noise with equal variance), you need an N 2 times bigger to have the same standard error of the mean.</p>

CC BY-SA 3.0
2015-03-22T17:09:43.283Z
+1. Very interesting answer, but it doesn't quite satisfy as is. That there is reason to suspect the tradeoffs between fidelity and quantity is given in the question, but it is conceivable that the cost is in fact so small or constant it can be statistically controlled for or ignored.
Christian Hummeluhr
CC BY-SA 3.0
2015-03-22T17:51:31.033Z
@ChristianHummeluhr I am not really sure what you mean, but I added some more about jitter.
StrongBad
CC BY-SA 3.0
2015-03-22T19:28:11.157Z
Thank you. While I am a sucker for an EJW citation, I'm afraid that I am not sure what this paper has to do with the question. The paper is about the relationship of the mean to the variance in RT data, and does not mention jitter or compare online to on-site data. Is the link correct?
Christian Hummeluhr
CC BY-SA 3.0
2015-03-22T19:37:50.830Z
@ChristianHummeluhr the EJW citation tells that the variance/jitter of optimally collected RT data is large enough that the additional variance/jitter from online data collection does not matter.
StrongBad
CC BY-SA 3.0
2015-03-22T16:38:43.870Z
2015-03-22T19:39:33.660Z
2015-03-22T19:39:33.660Z
StrongBad
<p>Online studies promise the possibility of greatly increased numbers of and variability in populations to study, but there are many potential concerns and need for validation, and diving in head first seems imprudent. </p>

<p>Here, I am interested in the ability to collect <strong>response time</strong> data online compared to a standard computer setup (e.g., a PsychToolbox or E-Prime based study with keyboard input, not a response box) in an on-site experiment booth. While these studies have their own limitations, such as being insuitable for experiments where very high fidelity RT data is critical to the study, I am merely interested in whether online studies differ significantly from them.</p>

<ul>
<li><strong>Are online RTs different from on-site RTs within tasks?</strong></li>
<li><strong>Is the theoretical floor on online RTs in general different from that of on-site RTs?</strong></li>
</ul>

CC BY-SA 3.0
2015-03-22T12:23:38.457Z
Does this answer your question?  http://cogsci.stackexchange.com/questions/109/open-source-software-for-running-internet-psychological-experiments-that-collect/9487#9487
AliceD
CC BY-SA 3.0
2015-03-22T12:26:07.163Z
Are you interested in millisecond range data? Or seconds range? In the millisecond range internet and reliance on two systems + servers will be killing
AliceD
CC BY-SA 3.0
2015-03-22T12:32:28.673Z
It doesn't, but the citation seems useful if I eventually answer this myself. The question is methodological, so I don't have a particular range in mind--I'm interested in how online studies compare to on-site studies on different tasks, and at what point online studies start to lose fidelity in general, and whether that point is different from that for keyboard based on-site methods.
Christian Hummeluhr
CC BY-SA 3.0
2015-03-22T12:36:37.687Z
I'm also not particularly interested in response box comparisons (but not totally uninterested), because most people don't have a response box at home, so there's little benefit to be had there in any case.
Christian Hummeluhr
<measurement><experimental-psychology><methodology><internet>
How valid are reaction times collected from online studies?
Christian Hummeluhr
# How valid are reaction times collected from online studies?

(Asked by: username_0 on 2015-03-22)

Online studies promise the possibility of greatly increased numbers of and variability in populations to study, but there are many potential concerns and need for validation, and diving in head first seems imprudent.

Here, I am interested in the ability to collect **response time** data online compared to a standard computer setup (e.g., a PsychToolbox or E-Prime based study with keyboard input, not a response box) in an on-site experiment booth. While these studies have their own limitations, such as being insuitable for experiments where very high fidelity RT data is critical to the study, I am merely interested in whether online studies differ significantly from them.

* **Are online RTs different from on-site RTs within tasks?**
* **Is the theoretical floor on online RTs in general different from that of on-site RTs?**

**Tags:** measurement, experimental-psychology, methodology, internet

> **Comments:**
> **username_1**: Does this answer your question?  http://cogsci.stackexchange.com/questions/109/open-source-software-for-running-internet-psychological-experiments-that-collect/9487#9487  
> **username_1**: Are you interested in millisecond range data? Or seconds range? In the millisecond range internet and reliance on two systems + servers will be killing  
> **username_0**: It doesn't, but the citation seems useful if I eventually answer this myself. The question is methodological, so I don't have a particular range in mind--I'm interested in how online studies compare to on-site studies on different tasks, and at what point online studies start to lose fidelity in general, and whether that point is different from that for keyboard based on-site methods.  
> **username_0**: I'm also not particularly interested in response box comparisons (but not totally uninterested), because most people don't have a response box at home, so there's little benefit to be had there in any case.  

---

## 3 answers

### Answer by username_2 on 2015-03-22. Score: 7

There are a few factors that could contribute to differences between online versus in-lab reaction time measurement.

**Hardware variation**

Participants in an online experiment will use their own computers to complete the task, which will result in lots of variation in hardware. Many studies have looked at how hardware variations affect response time measurement, and generally find that hardware variations can cause differences in the range of 10-100ms for a single response (e.g. [Plant & Turner, 2009](http://www.ncbi.nlm.nih.gov/pubmed/19587169)).

**Software variation**

Online studies and lab studies tend to be conducted with different software, since most of the standard lab-based software can't be used to make an online experiment. A popular option for online experiments is JavaScript & HTML. [Reimers & Stewart (2014)](http://www.ncbi.nlm.nih.gov/pubmed/24903687) measured the error in JavaScript response time measurements, and generally found it to be around 25ms, with some variation across different hardware and software. [de Leeuw & Motz (2015)](http://link.springer.com/article/10.3758/s13428-015-0567-2) ran an experiment in which subjects completed a visual search task in the lab using both a JavaScript and MATLAB (Psychophysics Toolbox) version of the experiment, and found that JavaScript measured response times that were about 25ms slower. However, both JavaScript and MATLAB had equivalent variance in the measurements and both software systems were equally sensitive to experimental manipulations of the visual search task at the sample sizes used for the experiment.

**Online versus in the lab**

[Hilbig (in press)](http://dx.doi.org/10.3758/s13428-015-0678-9) randomly assigned participants to complete an experiment in the lab using standard lab software (E-prime), in the lab using a web browser, or online at a location of the participant's choice. They measured response times in a standard lexical decision task, and found that there were no significant differences among the three groups. The effect was on the order of 120-220ms. The effect is relatively large (d' ~ 1.5), but given the current literature there is no reason to doubt that it would.

**Does it matter?**

The last part of the answer is: does it really matter if response times collected online are noisier than those collected in the lab? It turns out that even for pretty noisy measurements, moderate sample sizes will counteract the additional noise of the measurement. Reimers & Stewart (2014) simulated the sample size necessary to find an effect of 50ms with and without the additional noise in response times caused by using online methods. They found that only about 10% more subjects were needed in their model to have an equivalent likelihood of detecting the effect. [Ulrich & Giray (1989)](http://onlinelibrary.wiley.com/doi/10.1111/j.2044-8317.1989.tb01111.x/full) came to a similar conclusion in a different modeling context.

**References**

* de Leeuw, J. R., & Motz, B. A. (2015). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. *Behavior Research Methods*. [doi:10.3758/s13428-015-0567-2](http://dx.doi.org/10.3758/s13428-015-0567-2)
* Hilbig, B. E. (in press). Reaction time effects in lab- versus web-based research: Experimental evidence. *Behavior Research Methods*. [doi:10.3758/s13428-015-0678-9](http://dx.doi.org/10.3758/s13428-015-0678-9)
* Plant, R., & Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: new hardware, new problems? *Behavior Research Methods*, *41*(3), 598-614.
* Reimers, S., & Stewart, N. (2014). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. *Behavior Research Methods*
* Ulrich, R., & Giray, M. (1989). Time resolution of clocks: Effects on reaction time measurement - Good news for bad clocks. *British Journal of Mathematical and Statistical Psychology*, *42*, 1-12.

> **Comments:**
> **username_0**: I'm currently out of votes to up this answer, but I wanted to give a special thanks for sharing your summary of the unpublished data with just the right details (which I will absolutely treat with utmost suspicion until it has been reviewed by three arbitrary persons and locked securely behind an Elsevier paywall). I look forward to reading your paper!  

---

### Answer by username_3 on 2015-05-05. Score: 3

We cover a discussion on this in an article we've submitted for peer-review. Here is the [preprint](https://peerj.com/preprints/921/).

I will cite this stackExchange question/answers in the manuscript (post peer review now) as there are some lovely discussions going on, and doubtless, more to follow.

Tangentially relevant to this discussion is a simulation we did in the paper exploring how not knowing screen refresh impacts on stimulus timing (consider that if RT *starts* being recorded from a stimulus presentation, error in stimulus timing mucks around with RT):

> We tested this appearance issue in a simulation where we varied the
> duration of visual stimulus, starting at a random time during the
> refresh cycle (10,000 virtual presentations per stimulus duration).
> Figure 5 shows the likelihood of short duration stimuli being shown at
> all, or being shown for the wrong duration, or starting / stopping at
> the wrong time (<https://github.com/andytwoods/refreshSimulation>;
> available to run / tweak online here
> <http://jsfiddle.net/andytwoods/0f56hmaf/>).
> ![enter image description here](https://i.sstatic.net/2RctD.png)

Below is the abstract:

> This article provides an overview of the literature on the use of
> internet-based testing to address questions in perception research.
> Internet-based testing has several advantages over in-lab research,
> including the ability to reach a relatively broad set of participants
> and to quickly and inexpensively collect large amounts of empirical
> data. In many cases, the quality of online data appears to match that
> collected in laboratory research. Generally speaking, online
> participants tend to be more representative of the population at large
> than laboratory based participants. There are, though, some important
> caveats, when it comes to collecting data online. It is obviously much
> more difficult to control the exact parameters of stimulus
> presentation (such as display characteristics) in online research.
> There are also some thorny ethical considerations that need to be
> considered by experimenters. Strengths and weaknesses of the online
> approach, relative to others, are highlighted, and recommendations
> made for those researchers who might be thinking about conducting
> their own studies using this increasingly-popular approach to research
> in the psychological sciences.

> **Comments:**
> **username_4**: Welcome to CogSci Andy! That would great! To improve this answer, would you mind adding a short synopsis of your recent publication?  
> **username_0**: I  can't tell you how happy I am that you found useful information for your review here (and about you submitting it for preprint). With all the power problems we have in the field, knowing when and how much we can trust online studies will make a huge difference. Based on a skim, it looks like I'll be reading a lot of the papers you cite.  
> **username_3**: hope that's sufficient.  

---

### Answer by username_5 on 2015-03-22. Score: 1

Depending on how you collect the data, reaction times collected "online" will likely be different from those collected "on-site". When considering reaction times, it is important to decide if the reaction time is being used as a trigger, as the time to a response, or the difference in the time to response.

Consider an experiment which displays a random series of images for a 1/2 second each and the analysis consists of averaging the images that resulted in key presses. If your online system introduces a 1 s delay, you will not be averaging the images that led to the keypress, but the random image after it.

Consider an experiment which displays a random series of images for a 1/2 second each and every once in a while a target image is displayed and the analysis focuses on average amount of time it takes to react to the target image. In this case your reaction time will be 1 second longer than it should be and your data will be effectively meaningless.

Consider an experiment which displays a random series of images for a 1/2 second each and every once in a while one of two target images is displayed and the analysis focuses on difference in the average amount of time it takes to react to the target image. In this case the reaction time to each target will be 1 second longer than it should be, but the difference in reaction will be accurate. If in this experiment, in addition to the 1 s delay, there is also a variable delay (e.g., a Gaussian jitter with mean 0 and variance 1 s). This jitter will add noise to the data and make seeing small difference difficult. This noise, however, will average out across the trials and participants.

As with most psychological measures, the experimenter can trade fidelity of the measurement, number of measurements on each participant, and number of participants against each other. Online studies give up fidelity, and to an extent number of measurements on each participant, for large numbers of participants.

In a lab with a dedicated button box on fast dedicated hardware response latencies of under 1 ms are possible with tiny jitters. This of course ignores the human subject making the response. [Wagenmakers et al (2005)](http://www.sciencedirect.com/science/article/pii/S0022249605000106) suggest jitter of about 100 ms in the best case. Adding keyboard and network jitter of 100 ms (which would be pretty bad) means the online experiment would need 2x increase in the number of subjects to have the same statistical power; if the variance is twice as big (which happens when you add two independent sources of noise with equal variance), you need an N 2 times bigger to have the same standard error of the mean.

> **Comments:**
> **username_0**: +1. Very interesting answer, but it doesn't quite satisfy as is. That there is reason to suspect the tradeoffs between fidelity and quantity is given in the question, but it is conceivable that the cost is in fact so small or constant it can be statistically controlled for or ignored.  
> **username_5**: @ChristianHummeluhr I am not really sure what you mean, but I added some more about jitter.  
> **username_0**: Thank you. While I am a sucker for an EJW citation, I'm afraid that I am not sure what this paper has to do with the question. The paper is about the relationship of the mean to the variance in RT data, and does not mention jitter or compare online to on-site data. Is the link correct?  
> **username_5**: @ChristianHummeluhr the EJW citation tells that the variance/jitter of optimally collected RT data is large enough that the additional variance/jitter from online data collection does not matter.  

---

