<p>A pediatric neuropsychologist friend of mine was able to find two more studies that empirically measure the rate at which other studies succeed or fail at replicating.</p>
<p>&quot;Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015&quot;, Colin F Camerer et al, 2018, <a href="https://pubmed.ncbi.nlm.nih.gov/31346273/" rel="nofollow noreferrer">https://pubmed.ncbi.nlm.nih.gov/31346273/</a></p>
<blockquote>
<p>We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. [...] Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis.</p>
</blockquote>
<p>&quot;Many Labs 2: Investigating Variation in Replicability Across Samples and Settings&quot;, Richard A. Klein et al, 2018, <a href="https://journals.sagepub.com/doi/10.1177/2515245918810225" rel="nofollow noreferrer">https://journals.sagepub.com/doi/10.1177/2515245918810225</a></p>
<blockquote>
<p>We conducted preregistered replications of 28 classic and contemporary published findings [...] Using the conventional criterion of statistical significance (p &lt; .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding.</p>
</blockquote>
<p>The original OSC paper said &quot;There is no single standard for evaluating replication success&quot;, but gave figures ranging from 36% to 39% depending on what metrics were used to define &quot;replication success&quot;.</p>
<p>These follow up studies are a bit more optimistic, providing figures ranging from 54% to 67%</p>
<p>I would love to receive more examples of studies which empirically attempted to replicate previously performed studies to help us get a better idea of what the base rate probability is for a given study to replicate.</p>

CC BY-SA 4.0
2024-04-14T00:36:58.950Z
2024-04-14T00:36:58.950Z
Nebu Pookins
<p>I've only been able to find two studies published in Science on this:</p>
<ul>
<li><a href="https://www.science.org/doi/full/10.1126/science.aac4716" rel="nofollow noreferrer">Estimating the reproducibility of psychological science&quot;, Open Science Collaboration (OSC), 2015</a></li>
<li><a href="https://www.science.org/doi/10.1126/science.aad7243" rel="nofollow noreferrer">Comment on 'Estimating the reproducibility of psychological science, by Gilbert et al, 2016</a></li>
</ul>
<p>The OSC paper essentially says &quot;yes&quot;. The Gilbert paper essentially says that the OSC paper is flawed and provides no evidence that the answer is yes (Gilbert et al do not assert whether the answer to my question is &quot;yes&quot; or &quot;no&quot;, but rather that we should not accept the OSC paper as evidence for &quot;yes&quot;).</p>
<p>Maybe my google-fu isn't very good, but when I try to search for follow up studies on this topic, I only find popular articles reporting on the OSC study, giving me the impression that the general consensus is &quot;yes&quot;, but with the caveat that the general consensus might not be aware of Gilbert's rebuttal.</p>
<p><strong>What follow up studies have been done on this topic?</strong></p>
<p>I'm not interested in theoretical arguments for why a replication crisis may be more or less likely (e.g. arguments of the form &quot;publication bias makes it so that researchers are more likely to publish papers if their experiment lead to a positive result&quot;, etc.). I'm looking for empirical studies that examine what proportion of studies in psychology (or other domains of science) fail to replicate.</p>
<h1>Update 2024-April-11:</h1>
<p>I'll try to clarify my problems discovering relevant studies. As per the suggestion from @AliceD, I went to Google Scholar and searched for &quot;replication crisis&quot; with papers published in 2016 or later.</p>
<h2>First Paper</h2>
<p>The first result for me is &quot;Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis&quot;, 2018, PE Shrout, <a href="https://www.annualreviews.org/content/journals/10.1146/annurev-psych-122216-011845" rel="nofollow noreferrer">https://www.annualreviews.org/content/journals/10.1146/annurev-psych-122216-011845</a></p>
<p>This paper opens with &quot;Why do people say there is a crisis?&quot;, and says there are &quot;three sets of events&quot; that led to this. The first is a couple of cases of scientific fraud. I suppose this &quot;sort of&quot; gives us a lower-bound for the raw count of studies that failed to replicate (although not really, because it's plausible that a fraudulent study might describe an effect that does actually replicate!!). Even assuming that the fraudulent studies did not replicate, it would be hard to take a list of examples and extrapolate what proportion of studies in general do not replicate.</p>
<p>The second event that PE Shrout mentions is &quot;the publication of articles by a series of authors (Ioannidis 2005, Kerr 1998, Simmons et al. 2011, Vul et al. 2009) criticizing questionable research practices&quot;. This to me is akin to the theoretical arguments that I am not particularly interested in. The Ioannidis 2005 paper does not &quot;go out&quot; and empirically measure how many studies failed to replicate, but rather constructs mathematical models and simulations, showing the relationship between pre-study odds, the power of the study, bias, and the post-study probability. Simmons et al. 2011 similarly uses computer simulation as the main argument in their study.</p>
<p>I was not able to get access to the Kerr 1998 article, but from the abstract, it sounds like it mainly discusses HARKing (Hypothesizing After the Results are Known), why scientists might HARK, and proposals to avoid HARKing. Again, while interesting, it does not sound like this would lead to an estimate on the proportion of studies that would not replicate.</p>
<p>I also cannot get access to the Vul et al. 2009 study, and it seems to focus on a specific type of mistake found in fMRI studies (&quot;a strategy that computes separate correlations for individual voxels and reports means of only those voxels exceeding chosen thresholds. We show how this nonindependent analysis inflates correlations while yielding reassuring-looking scattergrams.&quot;) Again, interesting, but it's unclear to me how to compute base-rate of probability of replication from this.</p>
<p>The third event that PE Shrout mentions is the OSC paper I mentioned in my original post.</p>
<p>Maybe I missed it in my skim of PE Shrout's paper, but they do not seem to be aware of Gilbert's rebuttal to the OSC paper, and they take the OSC's result (that there IS a replication crisis) as given, and proceed to look for problems in how psychology research is performed and what can be done to address these problems.</p>
<h2>Second paper</h2>
<p>The second result for me is &quot;The replication crisis in psychology: An overview for theoretical and philosophical psychology.&quot;, 2019, BJ Wiggins <a href="https://psycnet.apa.org/record/2019-58388-002" rel="nofollow noreferrer">https://psycnet.apa.org/record/2019-58388-002</a></p>
<p>I'm not able to access this paper, but from the abstract, it sounds like the paper accepts it as a given that there is a replication crisis, and dives straight into arguing &quot;that the problems of the replication crisis and the concerns of the reform movement in fact provide various points of entry for theoretical and philosophical psychologists to collaborate with reformers in providing a more deeply philosophical critique and reform.&quot;</p>
<h2>Third paper</h2>
<p>The third result for me is &quot;Psychology's Replication Crisis and Clinical Psychological Science&quot;, 2019, JL Tackett <a href="https://www.annualreviews.org/content/journals/10.1146/annurev-clinpsy-050718-095710" rel="nofollow noreferrer">https://www.annualreviews.org/content/journals/10.1146/annurev-clinpsy-050718-095710</a></p>
<p>This paper also cites the OSC study I originally mentioned. I was a little excited when I saw the authors state:</p>
<blockquote>
<p>Differences in methodology and statistical approaches [e.g., an emphasis on effect size in clinical research over focal hypotheses examined via null hypothesis significance testing (NHST) in experimental social and cognitive psychology] may point to potential differences in overall replication of findings, but until this issue is explicitly and comprehensively examined, it remains an empirical question (Tackett et al. 2017)</p>
</blockquote>
<p>But my hopes were dashed when I read the abstract of the Tackett et al. 2017 paper. I was not able to get access to the paper, but based on the abstract, it seems like (Tackett et al. 2017) once again take the replication crisis as a given, and then &quot;(a) examine why clinical psychology and allied fields, such as counseling and school psychology, have not been central participants in the replicability conversation; (b) review concerns and recommendations that are less (or more) applicable to or appropriate for research in clinical psychology and allied fields; and (c) generate take-home messages for scholars and consumers of the literature in clinical psychology and allied fields, as well as reviewers, editors, and colleagues from other areas of psychological science.&quot;</p>
<p>Going back to the JL Tackett 2019 paper, it seems to also take the OSC result as given, and tries to find reasons for why psychology might be especially vulnerable to the problem (e.g. &quot;Historically, statistical power has been poor in psychological research&quot;, &quot;Interrater reliability for diagnoses has historically been mixed at best&quot;, &quot;the impact of publication bias in clinical science&quot;, etc.) Again, these are theoretical arguments for why we're likely to have a replication crisis, but not an empirical measure of the size of the crisis.</p>
<h2>Summary</h2>
<p>This is my general experience when looking into the issue myself. Almost all papers cite OSC, and do <em>not</em> cite Gilbert's rebuttal to OSC. The papers then take the replication crisis as given and then expand on that.</p>
<p>I want to take Gilbert's rebuttal seriously, and do not want to take the replication crisis as given. I would like to take a step back and ask &quot;wait, do we even actually have a replication crisis?&quot;</p>
<p>I guess operationally, I would like some framework for answering the question &quot;Before reading this psychology paper, what is the prior probability that the result of the paper will replicate?&quot; and I think one of the key piece of data we'd need to answer that question is what the base rate is for psychology papers to replicate (perhaps conditional on when it was published, or other factors). I feel like this is best answered empirically, by examining a large group of papers and seeing how many of them replicated.</p>
<p>In other words, I think I want a repeat of what OSC did, but that addresses Gilbert's criticisms.</p>

CC BY-SA 4.0
2024-04-11T08:49:32.713Z
A Google Scholar search using "replication" "crisis" and restricting the query to papers published 2016 and onwards yielded many hits; skimming through the first 20 pages I saw that most of the papers were relevant, so I reckon at least 200 papers address the problem and likely many more; maybe not all specifically oriented on CogSci, but problems in other domains are often a signal for a science wide issue. Physics is a discipline that tried to break free from this by enforcing impactful results to be published as significant only when p is < 0.0001 or something.
AliceD
CC BY-SA 4.0
2024-04-11T08:52:47.927Z
I -1d your post, because the answer seems a bit too obvious. Interesting topic though and I upvoted at first because there was at least some research effort that showed in your post, making the question actually pretty good. Perhaps you can specify the question further and explain what you wish to learn from this exactly.
AliceD
CC BY-SA 4.0
2024-04-11T17:10:50.083Z
Thank you for your response @AliceD I've updated the question to try to clarify exactly what I wish to learn.
Nebu Pookins
CC BY-SA 4.0
2024-04-12T07:59:22.827Z
Great job; but...realize that many people are busy and fill up slivers of their time answering Qs. Many will hence abandon posts that are too wordy; I for instance do not have the time to read through all the material above and won't dive into it, although I would like to; this is simply too much work. Best practice is hence to keep things information-dense and compact. FYI, I removed my -1 as you improved the post a lot; yet due to reasons above I fear that it will receive less attention than it really deserves. I recommend editing it to make it short and sweet.
AliceD
CC BY-SA 4.0
2024-04-12T13:47:59.020Z
There is so much written on the replication crisis in psychology that I'm confused that you're having trouble finding examples. Perhaps that's because you're looking for quantitative evidence of something that is inherently subjective and qualitative. "Replication crisis" is not something you can put a number on. I think if you're looking for numbers, you might instead look at measures of things that might *contribute to* a replication crisis, such as studies of publication bias that look at the distribution of p-values in published work.
Bryan Krause
CC BY-SA 4.0
2024-04-12T13:49:55.767Z
It's important to realize that evidence of a replication crisis does not necessarily come from identifying specific papers that fail to replicate. If so, we wouldn't have a crisis, we could just identify those papers and toss them out and be happy with the rest. The nature of the crisis is the **inability to identify which papers have the wrong conclusions**, which undermines confidence in a much larger pool of papers. Some are correct, some are wrong; the problem is we can't identify which are which without substantial investment in redoing the studies.
Bryan Krause
CC BY-SA 4.0
2024-04-14T00:25:20.340Z
@BryanKrause thank you for your response. I believe that your aforementioned "substantial investment in redoing the studies" is one of (many of) the efforts we need to address the replication crisis. There needs to be a cultural change or an incentives shift so that our default instinct is to want to see replication studies done before we're confident a proposed effect is true.

The OSC study is one such example of a substantial investement in redoing the studies. I am asking if people are aware of other examples.
Nebu Pookins
CC BY-SA 4.0
2024-04-14T07:43:41.597Z
"Almost all papers cite OSC, and do not cite Gilbert's rebuttal to OSC." Have you gone [through citations of the latter](https://scholar.google.com/scholar?cites=9725423855580250490&as_sdt=2005&sciodt=0,5&hl=en)?
Steven Jeuris
<meta-analysis>
Is there a replication crisis in psychology? How do we know?
Nebu Pookins
# Is there a replication crisis in psychology? How do we know?

(Asked by: username_0 on 2024-04-11)

I've only been able to find two studies published in Science on this:

* [Estimating the reproducibility of psychological science", Open Science Collaboration (OSC), 2015](https://www.science.org/doi/full/10.1126/science.aac4716)
* [Comment on 'Estimating the reproducibility of psychological science, by Gilbert et al, 2016](https://www.science.org/doi/10.1126/science.aad7243)

The OSC paper essentially says "yes". The Gilbert paper essentially says that the OSC paper is flawed and provides no evidence that the answer is yes (Gilbert et al do not assert whether the answer to my question is "yes" or "no", but rather that we should not accept the OSC paper as evidence for "yes").

Maybe my google-fu isn't very good, but when I try to search for follow up studies on this topic, I only find popular articles reporting on the OSC study, giving me the impression that the general consensus is "yes", but with the caveat that the general consensus might not be aware of Gilbert's rebuttal.

**What follow up studies have been done on this topic?**

I'm not interested in theoretical arguments for why a replication crisis may be more or less likely (e.g. arguments of the form "publication bias makes it so that researchers are more likely to publish papers if their experiment lead to a positive result", etc.). I'm looking for empirical studies that examine what proportion of studies in psychology (or other domains of science) fail to replicate.

# Update 2024-April-11:

I'll try to clarify my problems discovering relevant studies. As per the suggestion from @username_1, I went to Google Scholar and searched for "replication crisis" with papers published in 2016 or later.

## First Paper

The first result for me is "Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis", 2018, PE Shrout, <https://www.annualreviews.org/content/journals/10.1146/annurev-psych-122216-011845>

This paper opens with "Why do people say there is a crisis?", and says there are "three sets of events" that led to this. The first is a couple of cases of scientific fraud. I suppose this "sort of" gives us a lower-bound for the raw count of studies that failed to replicate (although not really, because it's plausible that a fraudulent study might describe an effect that does actually replicate!!). Even assuming that the fraudulent studies did not replicate, it would be hard to take a list of examples and extrapolate what proportion of studies in general do not replicate.

The second event that PE Shrout mentions is "the publication of articles by a series of authors (Ioannidis 2005, Kerr 1998, Simmons et al. 2011, Vul et al. 2009) criticizing questionable research practices". This to me is akin to the theoretical arguments that I am not particularly interested in. The Ioannidis 2005 paper does not "go out" and empirically measure how many studies failed to replicate, but rather constructs mathematical models and simulations, showing the relationship between pre-study odds, the power of the study, bias, and the post-study probability. Simmons et al. 2011 similarly uses computer simulation as the main argument in their study.

I was not able to get access to the Kerr 1998 article, but from the abstract, it sounds like it mainly discusses HARKing (Hypothesizing After the Results are Known), why scientists might HARK, and proposals to avoid HARKing. Again, while interesting, it does not sound like this would lead to an estimate on the proportion of studies that would not replicate.

I also cannot get access to the Vul et al. 2009 study, and it seems to focus on a specific type of mistake found in fMRI studies ("a strategy that computes separate correlations for individual voxels and reports means of only those voxels exceeding chosen thresholds. We show how this nonindependent analysis inflates correlations while yielding reassuring-looking scattergrams.") Again, interesting, but it's unclear to me how to compute base-rate of probability of replication from this.

The third event that PE Shrout mentions is the OSC paper I mentioned in my original post.

Maybe I missed it in my skim of PE Shrout's paper, but they do not seem to be aware of Gilbert's rebuttal to the OSC paper, and they take the OSC's result (that there IS a replication crisis) as given, and proceed to look for problems in how psychology research is performed and what can be done to address these problems.

## Second paper

The second result for me is "The replication crisis in psychology: An overview for theoretical and philosophical psychology.", 2019, BJ Wiggins <https://psycnet.apa.org/record/2019-58388-002>

I'm not able to access this paper, but from the abstract, it sounds like the paper accepts it as a given that there is a replication crisis, and dives straight into arguing "that the problems of the replication crisis and the concerns of the reform movement in fact provide various points of entry for theoretical and philosophical psychologists to collaborate with reformers in providing a more deeply philosophical critique and reform."

## Third paper

The third result for me is "Psychology's Replication Crisis and Clinical Psychological Science", 2019, JL Tackett <https://www.annualreviews.org/content/journals/10.1146/annurev-clinpsy-050718-095710>

This paper also cites the OSC study I originally mentioned. I was a little excited when I saw the authors state:

> Differences in methodology and statistical approaches [e.g., an emphasis on effect size in clinical research over focal hypotheses examined via null hypothesis significance testing (NHST) in experimental social and cognitive psychology] may point to potential differences in overall replication of findings, but until this issue is explicitly and comprehensively examined, it remains an empirical question (Tackett et al. 2017)

But my hopes were dashed when I read the abstract of the Tackett et al. 2017 paper. I was not able to get access to the paper, but based on the abstract, it seems like (Tackett et al. 2017) once again take the replication crisis as a given, and then "(a) examine why clinical psychology and allied fields, such as counseling and school psychology, have not been central participants in the replicability conversation; (b) review concerns and recommendations that are less (or more) applicable to or appropriate for research in clinical psychology and allied fields; and (c) generate take-home messages for scholars and consumers of the literature in clinical psychology and allied fields, as well as reviewers, editors, and colleagues from other areas of psychological science."

Going back to the JL Tackett 2019 paper, it seems to also take the OSC result as given, and tries to find reasons for why psychology might be especially vulnerable to the problem (e.g. "Historically, statistical power has been poor in psychological research", "Interrater reliability for diagnoses has historically been mixed at best", "the impact of publication bias in clinical science", etc.) Again, these are theoretical arguments for why we're likely to have a replication crisis, but not an empirical measure of the size of the crisis.

## Summary

This is my general experience when looking into the issue myself. Almost all papers cite OSC, and do *not* cite Gilbert's rebuttal to OSC. The papers then take the replication crisis as given and then expand on that.

I want to take Gilbert's rebuttal seriously, and do not want to take the replication crisis as given. I would like to take a step back and ask "wait, do we even actually have a replication crisis?"

I guess operationally, I would like some framework for answering the question "Before reading this psychology paper, what is the prior probability that the result of the paper will replicate?" and I think one of the key piece of data we'd need to answer that question is what the base rate is for psychology papers to replicate (perhaps conditional on when it was published, or other factors). I feel like this is best answered empirically, by examining a large group of papers and seeing how many of them replicated.

In other words, I think I want a repeat of what OSC did, but that addresses Gilbert's criticisms.

**Tags:** meta-analysis

> **Comments:**
> **username_1**: A Google Scholar search using "replication" "crisis" and restricting the query to papers published 2016 and onwards yielded many hits; skimming through the first 20 pages I saw that most of the papers were relevant, so I reckon at least 200 papers address the problem and likely many more; maybe not all specifically oriented on CogSci, but problems in other domains are often a signal for a science wide issue. Physics is a discipline that tried to break free from this by enforcing impactful results to be published as significant only when p is < 0.0001 or something.  
> **username_1**: I -1d your post, because the answer seems a bit too obvious. Interesting topic though and I upvoted at first because there was at least some research effort that showed in your post, making the question actually pretty good. Perhaps you can specify the question further and explain what you wish to learn from this exactly.  
> **username_0**: Thank you for your response @username_1 I've updated the question to try to clarify exactly what I wish to learn.  
> **username_1**: Great job; but...realize that many people are busy and fill up slivers of their time answering Qs. Many will hence abandon posts that are too wordy; I for instance do not have the time to read through all the material above and won't dive into it, although I would like to; this is simply too much work. Best practice is hence to keep things information-dense and compact. FYI, I removed my -1 as you improved the post a lot; yet due to reasons above I fear that it will receive less attention than it really deserves. I recommend editing it to make it short and sweet.  
> **username_2**: There is so much written on the replication crisis in psychology that I'm confused that you're having trouble finding examples. Perhaps that's because you're looking for quantitative evidence of something that is inherently subjective and qualitative. "Replication crisis" is not something you can put a number on. I think if you're looking for numbers, you might instead look at measures of things that might *contribute to* a replication crisis, such as studies of publication bias that look at the distribution of p-values in published work.  
> **username_2**: It's important to realize that evidence of a replication crisis does not necessarily come from identifying specific papers that fail to replicate. If so, we wouldn't have a crisis, we could just identify those papers and toss them out and be happy with the rest. The nature of the crisis is the **inability to identify which papers have the wrong conclusions**, which undermines confidence in a much larger pool of papers. Some are correct, some are wrong; the problem is we can't identify which are which without substantial investment in redoing the studies.  
> **username_0**: @BryanKrause thank you for your response. I believe that your aforementioned "substantial investment in redoing the studies" is one of (many of) the efforts we need to address the replication crisis. There needs to be a cultural change or an incentives shift so that our default instinct is to want to see replication studies done before we're confident a proposed effect is true.  The OSC study is one such example of a substantial investement in redoing the studies. I am asking if people are aware of other examples.  
> **username_3**: "Almost all papers cite OSC, and do not cite Gilbert's rebuttal to OSC." Have you gone [through citations of the latter](https://scholar.google.com/scholar?cites=9725423855580250490&as_sdt=2005&sciodt=0,5&hl=en)?  

---

## 1 answers

### Answer by username_0 on 2024-04-14. Score: 1

A pediatric neuropsychologist friend of mine was able to find two more studies that empirically measure the rate at which other studies succeed or fail at replicating.

"Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015", Colin F Camerer et al, 2018, <https://pubmed.ncbi.nlm.nih.gov/31346273/>

> We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015. [...] Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis.

"Many Labs 2: Investigating Variation in Replicability Across Samples and Settings", Richard A. Klein et al, 2018, <https://journals.sagepub.com/doi/10.1177/2515245918810225>

> We conducted preregistered replications of 28 classic and contemporary published findings [...] Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding.

The original OSC paper said "There is no single standard for evaluating replication success", but gave figures ranging from 36% to 39% depending on what metrics were used to define "replication success".

These follow up studies are a bit more optimistic, providing figures ranging from 54% to 67%

I would love to receive more examples of studies which empirically attempted to replicate previously performed studies to help us get a better idea of what the base rate probability is for a given study to replicate.


---

