<p>gjacob is correct that optional stopping is a common research degree of freedom, and one that has a considerable and unfortunate intuitive basis. Yet, depending on the context of your research, AliceD's concerns are also important.</p>

<p>There is, however, a middle ground between not checking at all, and p-hacking: sequential analysis. There is a Bayesian version of sequential analysis, which I can update if that's your statistical paradigm, but I'm assuming you're wanting to conduct interim analyses using null-hypothesis significance testing, so that's what I'll focus on here. Lakens (2014) provides a nice overview of this practice. In essence, you take the level of $\alpha$ you want to maintain over your "peeks" (e.g., $\alpha$ = .05), and distribute that total $\alpha$ over the number of peeks you want to take along your total sampling process. Then, if $p$ is lower than this distributed $\alpha$ at any of your peeks, you can reject the null at $\alpha$ = .05, and you won't have inflated your Type I error rate like you would have with generic optional stopping.</p>

<p>It's a <em>little</em> more complicated than how I'm presenting here--and there are a number of methods for distributing your total $\alpha$--but not by much. If you can wrap your head around a bonferroni correction, it's a very similar technique. </p>

<p>Lakens, D. (2014). Performing high‐powered studies efficiently with sequential analyses. <em>European Journal of Social Psychology</em>, 44(7), 701-710.</p>

CC BY-SA 3.0
2018-03-23T19:41:20.770Z
Welcome and thanks for the great answer.+1
AliceD
CC BY-SA 3.0
2018-03-23T19:44:54.463Z
Thanks! I've been lurking on the beta here for awhile :) Been hoping to encourage more replicability related questions being asked (and answered) here, so that we have searchable q&a's vs. endless repetitive social media questions. Glad to see these sorts of q's appearing here!
jsakaluk
CC BY-SA 3.0
2018-03-23T20:50:47.680Z
This type of question might also be suited for CrossValidated and, perhaps, even Academia. So I think the number of questions like this posted on this stack will remain to be small.  Nonetheless, I think it's an awesome question, as it can be targeted from many view points, as the diverse answers show. Good stuff here.
AliceD
CC BY-SA 3.0
2018-03-23T16:58:14.847Z
2018-03-23T16:58:14.847Z
jsakaluk
<p>This is an important question! This practice ("optional stopping" if you stop collecting data based on your early analyses, or "peeking" if you continue collecting data) is considered a bad idea nowadays. It's a "researcher degree of freedom"--a practice that, in the long run and averaged across the field, appears to (empirically) result in high false-positive rates. It's a form of exploratory analysis, and although EA isn't bad in and of itself, optional stopping/peeking can predispose researchers to chase significance for trends they see in their data, perhaps by selectively excluding certain observations, dropping their a priori hypotheses, ignoring their a priori power analyses, etc...</p>

<p>Instead, consider running a power analysis. (I recommend <a href="http://www.gpower.hhu.de/en.html" rel="nofollow noreferrer">G*Power</a>, which is freely downloadable). I advise to perform a power analysis <em>before</em> you start collecting data, determine the total N that you'll shoot for, and don't peek at your data until you've hit that. It's effectively "blinding" yourself, much in the way that medical researchers might use double-blind studies to ensure the reliability of their findings.</p>

<p>Check out this paper for a longer discussion of researcher degrees of freedom: <a href="http://journals.sagepub.com/doi/abs/10.1177/0956797611417632" rel="nofollow noreferrer">http://journals.sagepub.com/doi/abs/10.1177/0956797611417632</a></p>

CC BY-SA 3.0
2018-03-21T17:40:57.220Z
2018-03-22T10:39:17.463Z
2018-03-22T10:39:17.463Z
qjacob
<p><strong>Short answer</strong><br>
From an ethical standpoint, not including interim evaluations may be bad practice.</p>

<p><strong>Background</strong><br>
I will start off with a more extreme case than in your question example, just for illustrative purposes, namely that of a <em>clinical intervention study</em>. If it appears that the treatment group (say, experimental medicine <em>Y</em>, instead of the standard of care treatment <em>X</em>) features substantially, if not significantly, more cases of serious adverse advents, or even deaths, that may (or may not) be related to treatment <em>Y</em>, the <strong>ethical best thing to do</strong> is put the study on hold until things are sorted out. This, to prevent any possibility of further physical harm caused by the experimental treatment. This happens quite regularly and should be followed by a report paper stating the results and a discussion on the best way to proceed this research, if applicable.</p>

<p>In a more <em>experimental setting</em> it may also be <strong>ethically best practice</strong> to evaluate preliminary data, as possible experimental design flaws, unexpected results (weird outcomes or artifacts in, say, left-handed people) or confounding factors may become apparent, and timely adjustments to the experimental protocol can be made. Why is this ethically correct? Because you may be subjecting people to a flawed experimental paradigm and wasting many hours of otherwise more productive time.</p>

<p>The case provided by the other answerer, where the experiment is stopped, while prior statistical power analysis was done, <em>that</em> is malpractice. Conversely, adding more subjects <em>post hoc</em> based on 'near-significance' is also questionable practice. But this is more related to what you do with the experimental interim data. In my opinion, they should be critically evaluated, but not so much on effect size, but rather on feasibility, correctness and validity - basically to sanity-check the study proceedings.     </p>

CC BY-SA 3.0
2018-03-22T09:28:19.430Z
You and @qjacob provide some very good points. In fact, I did do a power analysis, which is what I meant when I wrote "until a sufficient sample size is reached", but I guess that could have been more clear. I am glad I am not completely on the wrong path then. Thanks a lot!
Mikkel Schöttner
CC BY-SA 3.0
2018-03-23T19:13:22.493Z
This is a good answer, and I might also point out that a properly-designed experimental method is not subject to bias on the part of the person executing the method. It should not be possible to affect future results based on your knowledge of past results. If it is, you have a methodology issue.
theMayer
CC BY-SA 3.0
2018-03-23T19:47:04.583Z
@rmayer06: to clarify, are you saying that optional stopping has no impact on Type I error rate (as gjacob suggests below) if an experiment is properly-designed? Don't want to mischaracterize your position before responding.
jsakaluk
CC BY-SA 3.0
2018-03-24T06:55:57.633Z
@jsakaluk- yes, that's what I'm saying. All studies have time between runs, so there is stopping that takes place automatically. Obviously if you stop for a long period of time, you might have other issues and variability creep in.
theMayer
CC BY-SA 3.0
2018-03-22T08:51:45.930Z
2018-03-23T19:16:20.610Z
2018-03-23T19:16:20.610Z
AliceD
<p>For a research internship I am running psychological experiments online. As it takes a while until the experiment is done (meaning that enough people participated, so that a sufficient sample size is reached), I could already look at the data and run a few analyses with that incomplete data set. This way I could already see a trend in what the final result might be.</p>

<p>Are there any methodological reasons that would speak against that? For example, am I biasing myself in this way? Or is there any other reason this could be considered bad research practice?</p>

<methodology><experimental-psychology><bias><statistics><data>
Should I look at the data of an experiment before the dataset is complete?
Mikkel Schöttner
# Should I look at the data of an experiment before the dataset is complete?

(Asked by: username_0 on 2018-03-21)

For a research internship I am running psychological experiments online. As it takes a while until the experiment is done (meaning that enough people participated, so that a sufficient sample size is reached), I could already look at the data and run a few analyses with that incomplete data set. This way I could already see a trend in what the final result might be.

Are there any methodological reasons that would speak against that? For example, am I biasing myself in this way? Or is there any other reason this could be considered bad research practice?

**Tags:** methodology, experimental-psychology, bias, statistics, data


---

## 3 answers

### Answer by username_2 on 2018-03-22. Score: 5 | Accepted answer

**Short answer**  
From an ethical standpoint, not including interim evaluations may be bad practice.

**Background**  
I will start off with a more extreme case than in your question example, just for illustrative purposes, namely that of a *clinical intervention study*. If it appears that the treatment group (say, experimental medicine *Y*, instead of the standard of care treatment *X*) features substantially, if not significantly, more cases of serious adverse advents, or even deaths, that may (or may not) be related to treatment *Y*, the **ethical best thing to do** is put the study on hold until things are sorted out. This, to prevent any possibility of further physical harm caused by the experimental treatment. This happens quite regularly and should be followed by a report paper stating the results and a discussion on the best way to proceed this research, if applicable.

In a more *experimental setting* it may also be **ethically best practice** to evaluate preliminary data, as possible experimental design flaws, unexpected results (weird outcomes or artifacts in, say, left-handed people) or confounding factors may become apparent, and timely adjustments to the experimental protocol can be made. Why is this ethically correct? Because you may be subjecting people to a flawed experimental paradigm and wasting many hours of otherwise more productive time.

The case provided by the other answerer, where the experiment is stopped, while prior statistical power analysis was done, *that* is malpractice. Conversely, adding more subjects *post hoc* based on 'near-significance' is also questionable practice. But this is more related to what you do with the experimental interim data. In my opinion, they should be critically evaluated, but not so much on effect size, but rather on feasibility, correctness and validity - basically to sanity-check the study proceedings.

> **Comments:**
> **username_0**: You and @username_3 provide some very good points. In fact, I did do a power analysis, which is what I meant when I wrote "until a sufficient sample size is reached", but I guess that could have been more clear. I am glad I am not completely on the wrong path then. Thanks a lot!  
> **username_4**: This is a good answer, and I might also point out that a properly-designed experimental method is not subject to bias on the part of the person executing the method. It should not be possible to affect future results based on your knowledge of past results. If it is, you have a methodology issue.  
> **username_1**: @rmayer06: to clarify, are you saying that optional stopping has no impact on Type I error rate (as gjacob suggests below) if an experiment is properly-designed? Don't want to mischaracterize your position before responding.  
> **username_4**: @username_1- yes, that's what I'm saying. All studies have time between runs, so there is stopping that takes place automatically. Obviously if you stop for a long period of time, you might have other issues and variability creep in.  

---

### Answer by username_1 on 2018-03-23. Score: 5

gjacob is correct that optional stopping is a common research degree of freedom, and one that has a considerable and unfortunate intuitive basis. Yet, depending on the context of your research, username_2's concerns are also important.

There is, however, a middle ground between not checking at all, and p-hacking: sequential analysis. There is a Bayesian version of sequential analysis, which I can update if that's your statistical paradigm, but I'm assuming you're wanting to conduct interim analyses using null-hypothesis significance testing, so that's what I'll focus on here. Lakens (2014) provides a nice overview of this practice. In essence, you take the level of $\alpha$ you want to maintain over your "peeks" (e.g., $\alpha$ = .05), and distribute that total $\alpha$ over the number of peeks you want to take along your total sampling process. Then, if $p$ is lower than this distributed $\alpha$ at any of your peeks, you can reject the null at $\alpha$ = .05, and you won't have inflated your Type I error rate like you would have with generic optional stopping.

It's a *little* more complicated than how I'm presenting here--and there are a number of methods for distributing your total $\alpha$--but not by much. If you can wrap your head around a bonferroni correction, it's a very similar technique.

Lakens, D. (2014). Performing high‐powered studies efficiently with sequential analyses. *European Journal of Social Psychology*, 44(7), 701-710.

> **Comments:**
> **username_2**: Welcome and thanks for the great answer.+1  
> **username_1**: Thanks! I've been lurking on the beta here for awhile :) Been hoping to encourage more replicability related questions being asked (and answered) here, so that we have searchable q&a's vs. endless repetitive social media questions. Glad to see these sorts of q's appearing here!  
> **username_2**: This type of question might also be suited for CrossValidated and, perhaps, even Academia. So I think the number of questions like this posted on this stack will remain to be small.  Nonetheless, I think it's an awesome question, as it can be targeted from many view points, as the diverse answers show. Good stuff here.  

---

### Answer by username_3 on 2018-03-21. Score: 3

This is an important question! This practice ("optional stopping" if you stop collecting data based on your early analyses, or "peeking" if you continue collecting data) is considered a bad idea nowadays. It's a "researcher degree of freedom"--a practice that, in the long run and averaged across the field, appears to (empirically) result in high false-positive rates. It's a form of exploratory analysis, and although EA isn't bad in and of itself, optional stopping/peeking can predispose researchers to chase significance for trends they see in their data, perhaps by selectively excluding certain observations, dropping their a priori hypotheses, ignoring their a priori power analyses, etc...

Instead, consider running a power analysis. (I recommend [G\*Power](http://www.gpower.hhu.de/en.html), which is freely downloadable). I advise to perform a power analysis *before* you start collecting data, determine the total N that you'll shoot for, and don't peek at your data until you've hit that. It's effectively "blinding" yourself, much in the way that medical researchers might use double-blind studies to ensure the reliability of their findings.

Check out this paper for a longer discussion of researcher degrees of freedom: <http://journals.sagepub.com/doi/abs/10.1177/0956797611417632>


---

