<p><strong>Update (2023)</strong>:  A lot has happened in 10 years. I have published a couple of open meta-analyses myself. Matthew Jané has put together the more comprehensive compilation of open science meta-analyses (which include links to mine and many others):
<a href="https://matthewbjane.github.io/opensynthesis/" rel="nofollow noreferrer">https://matthewbjane.github.io/opensynthesis/</a></p>
<p><strong>Old Answer (2014):</strong> I have started to see a fair bit of discussion about reproducible meta-analysis.</p>
<p>Tim Churches seems to have a github repository with a <a href="https://github.com/timchurches/meta-analyses" rel="nofollow noreferrer">few examples of meta-analyses in R</a>.</p>
<p>See in particular the public health example:</p>
<ul>
<li><a href="https://github.com/timchurches/meta-analyses/blob/master/benefits-of-reproducible-research/benefits-of-reproducible-research.Rmd" rel="nofollow noreferrer">RMarkdown source</a></li>
<li><a href="https://github.com/timchurches/meta-analyses/blob/master/benefits-of-reproducible-research/benefits-of-reproducible-research.md" rel="nofollow noreferrer">Formatted Markdown output</a></li>
</ul>

CC BY-SA 4.0
2014-08-27T03:38:42.017Z
2023-04-16T05:51:39.783Z
2023-04-16T05:51:39.783Z
Jeromy Anglim
<p>In reference to your first question, not a lot of published standards, I suggest some <a href="http://www.lyonsmorris.com/MetaA/appendix1.htm" rel="nofollow noreferrer">standard guideline on my site</a>, but that's just that, suggestions. You might want to check out <a href="http://www.cochrane.org/what-is-cochrane-evidence" rel="nofollow noreferrer">the Cochrane Collaboration</a> though they have produced a set of standards for use in health care related meta-analysis.</p>

<p>You also may want to check an early-ish study by Shadish et al (1997) as an example:</p>

<p>Shadish, William, R., Matt, George, E., Navarro, Ana, M., Single, Gregory, Crits-Christoph, Paul, Hazelrigg, Anthony, J., Lyons, Larry C., et al., (1997). Evidence that therapy works in clinically representative conditions. Journal of Consulting and Clinical Psychology, 65, 355-65.</p>

CC BY-SA 3.0
2018-02-09T01:46:11.437Z
Thanks. Your answer speaks to the actual method of doing meta-analysis, as opposed to sharing the sharing of underlying data or scripts that do various steps (e.g., transformations of original effect sizes; combining of effect sizes within studies, correction formulas, etc.). Any thoughts on the reproducibility aspect?
Jeromy Anglim
CC BY-SA 3.0
2018-02-09T20:36:49.913Z
Given the same data set and using the same conversion routines etc., you should get the same results. One potential issue is with rater reliability - variation due to  those scoring the studies having slightly differing interpretations of the study factors. You should always report inter rater reliability. Alternatively where a discrepancy is discovered the project lead should go over the discrepancies with both raters and resolve them on a case by case basis, and note that in the study report.
Larry C. Lyons
CC BY-SA 3.0
2018-02-09T20:40:30.707Z
As for sharing the study data that is a given. There are a number of sites on the web that you can use, such as my personal favourite Github. You should also publish as appendicies the list of studies used the meta-analysis, and your study rating questionnaire.

As for transformations those are pretty standard. Any good stats textbook will show you those. For the web based application I wrote, the conversion javaScripts are freely available.
Larry C. Lyons
CC BY-SA 3.0
2018-02-09T20:40:41.700Z
These script are based on the formulas found in the following


Abramowitz, M. and Stegun, I. (1964). Handbook of mathematical functions with formulas, graphs, and mathematical tables. Washington, DC: United States Government Printing Office.


Cohen, J. (1988). Statistical power analysis for the behavioral sciences. (2nd Ed). Hillsdale, NJ: Lawrence Erlbaum Associates.


Hunter, J. and Schmidt, F. (2015). Methods of meta-analysis: Correcting error and bias in research findings. 3fd Ed. Beverly Hills CA: Sage.
Larry C. Lyons
CC BY-SA 3.0
2018-02-09T20:49:26.907Z
The Cochrane Collaboration also details its methods and procedures and again their conversion and accumulation methods are publically available.

Here are a couple of other links to some of my work, first a web based paper that explains the methodology and conversion techniques used,
http://www.lyonsmorris.com/MetaA/index.htm


and the application based on the paper. 
http://www.lyonsmorris.com/ma1/index.cfm


and the javascript used to do the conversions from study statistics to a common metric
http://www.lyonsmorris.com/ma1/js/ESConversions.js

hth larry
Larry C. Lyons
CC BY-SA 3.0
2018-02-08T19:11:44.473Z
2018-02-08T19:44:40.080Z
2018-02-08T19:44:40.080Z
Larry C. Lyons
<p>I am a <a href="http://jeromyanglim.blogspot.com.au/2012/07/video-knitr-rmarkdown-rstudio.html" rel="nofollow noreferrer">big proponent</a>  of <a href="https://stats.stackexchange.com/questions/14999/how-are-we-defining-reproducible-research/15006#15006">reproducible data analysis</a>. In particular, I like when researchers share documents in formats like <a href="http://www.google.com.au/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;ved=0CE8QFjAA&amp;url=http://www.statistik.lmu.de/~leisch/Sweave/&amp;ei=l0cmUNGJK7G6iAe3n4DYBw&amp;usg=AFQjCNFEae6blgD42G_2sDvkOct41_qwrw&amp;sig2=nf4P4h6ivSqOgE4NED3Ltw" rel="nofollow noreferrer">Sweave</a> and <a href="http://yihui.name/knitr/" rel="nofollow noreferrer">knitr</a> which weave statistical output (e.g., text, tables, and graphs) into a statistical report. </p>

<p>I previously asked for <a href="https://stats.stackexchange.com/q/1980/183">examples of reproducible research of any kind on Stats.SE</a> and <a href="https://cogsci.stackexchange.com/q/893/29">for advocating articles in psychology</a>.
However, at present I'm particularly interested in complete examples of reproducible meta-analysis.</p>

<p>Meta-analyses involve a number of steps. Summary statistics and study information is extracted from source studies. Various transformations and steps are applied to the data (e.g., corrections for reliability, conversion from one statistic to another, etc.). Various models are tested; tables and graphs are produced. Some journals are requiring that researchers supply tables of data used (e.g., references used, summary statistics).  However, I can see meta-analysis as an area which could benefit from a more comprehensive reproducible approach: (1) it would permit greater inspection of specific methods used; (2) researchers could more easily build on the analyses used.</p>

<p>Thus, my questions:</p>

<ul>
<li><strong>Are there any complete examples of reproducible meta-analysis preferably in psychology or a related discipline?</strong></li>
<li><strong>Is there any published advocacy for reproducible meta-analysis?</strong></li>
</ul>

<reference-request><statistics><reproducible-research><meta-analysis>
Reproducible meta-analysis in psychology
Jeromy Anglim
# Reproducible meta-analysis in psychology

(Asked by: username_0 on 2012-08-11)

I am a [big proponent](http://jeromyanglim.blogspot.com.au/2012/07/video-knitr-rmarkdown-rstudio.html) of [reproducible data analysis](https://stats.stackexchange.com/questions/14999/how-are-we-defining-reproducible-research/15006#15006). In particular, I like when researchers share documents in formats like [Sweave](http://www.google.com.au/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CE8QFjAA&url=http://www.statistik.lmu.de/~leisch/Sweave/&ei=l0cmUNGJK7G6iAe3n4DYBw&usg=AFQjCNFEae6blgD42G_2sDvkOct41_qwrw&sig2=nf4P4h6ivSqOgE4NED3Ltw) and [knitr](http://yihui.name/knitr/) which weave statistical output (e.g., text, tables, and graphs) into a statistical report.

I previously asked for [examples of reproducible research of any kind on Stats.SE](https://stats.stackexchange.com/q/1980/183) and [for advocating articles in psychology](https://cogsci.stackexchange.com/q/893/29).
However, at present I'm particularly interested in complete examples of reproducible meta-analysis.

Meta-analyses involve a number of steps. Summary statistics and study information is extracted from source studies. Various transformations and steps are applied to the data (e.g., corrections for reliability, conversion from one statistic to another, etc.). Various models are tested; tables and graphs are produced. Some journals are requiring that researchers supply tables of data used (e.g., references used, summary statistics). However, I can see meta-analysis as an area which could benefit from a more comprehensive reproducible approach: (1) it would permit greater inspection of specific methods used; (2) researchers could more easily build on the analyses used.

Thus, my questions:

* **Are there any complete examples of reproducible meta-analysis preferably in psychology or a related discipline?**
* **Is there any published advocacy for reproducible meta-analysis?**

**Tags:** reference-request, statistics, reproducible-research, meta-analysis


---

## 2 answers

### Answer by username_0 on 2014-08-27. Score: 2

**Update (2023)**: A lot has happened in 10 years. I have published a couple of open meta-analyses myself. Matthew Jané has put together the more comprehensive compilation of open science meta-analyses (which include links to mine and many others):
<https://matthewbjane.github.io/opensynthesis/>

**Old Answer (2014):** I have started to see a fair bit of discussion about reproducible meta-analysis.

Tim Churches seems to have a github repository with a [few examples of meta-analyses in R](https://github.com/timchurches/meta-analyses).

See in particular the public health example:

* [RMarkdown source](https://github.com/timchurches/meta-analyses/blob/master/benefits-of-reproducible-research/benefits-of-reproducible-research.Rmd)
* [Formatted Markdown output](https://github.com/timchurches/meta-analyses/blob/master/benefits-of-reproducible-research/benefits-of-reproducible-research.md)


---

### Answer by username_1 on 2018-02-08. Score: 0

In reference to your first question, not a lot of published standards, I suggest some [standard guideline on my site](http://www.lyonsmorris.com/MetaA/appendix1.htm), but that's just that, suggestions. You might want to check out [the Cochrane Collaboration](http://www.cochrane.org/what-is-cochrane-evidence) though they have produced a set of standards for use in health care related meta-analysis.

You also may want to check an early-ish study by Shadish et al (1997) as an example:

Shadish, William, R., Matt, George, E., Navarro, Ana, M., Single, Gregory, Crits-Christoph, Paul, Hazelrigg, Anthony, J., Lyons, Larry C., et al., (1997). Evidence that therapy works in clinically representative conditions. Journal of Consulting and Clinical Psychology, 65, 355-65.

> **Comments:**
> **username_0**: Thanks. Your answer speaks to the actual method of doing meta-analysis, as opposed to sharing the sharing of underlying data or scripts that do various steps (e.g., transformations of original effect sizes; combining of effect sizes within studies, correction formulas, etc.). Any thoughts on the reproducibility aspect?  
> **username_1**: Given the same data set and using the same conversion routines etc., you should get the same results. One potential issue is with rater reliability - variation due to  those scoring the studies having slightly differing interpretations of the study factors. You should always report inter rater reliability. Alternatively where a discrepancy is discovered the project lead should go over the discrepancies with both raters and resolve them on a case by case basis, and note that in the study report.  
> **username_1**: As for sharing the study data that is a given. There are a number of sites on the web that you can use, such as my personal favourite Github. You should also publish as appendicies the list of studies used the meta-analysis, and your study rating questionnaire.  As for transformations those are pretty standard. Any good stats textbook will show you those. For the web based application I wrote, the conversion javaScripts are freely available.  
> **username_1**: These script are based on the formulas found in the following   Abramowitz, M. and Stegun, I. (1964). Handbook of mathematical functions with formulas, graphs, and mathematical tables. Washington, DC: United States Government Printing Office.   Cohen, J. (1988). Statistical power analysis for the behavioral sciences. (2nd Ed). Hillsdale, NJ: Lawrence Erlbaum Associates.   Hunter, J. and Schmidt, F. (2015). Methods of meta-analysis: Correcting error and bias in research findings. 3fd Ed. Beverly Hills CA: Sage.  
> **username_1**: The Cochrane Collaboration also details its methods and procedures and again their conversion and accumulation methods are publically available.  Here are a couple of other links to some of my work, first a web based paper that explains the methodology and conversion techniques used, http://www.lyonsmorris.com/MetaA/index.htm   and the application based on the paper.  http://www.lyonsmorris.com/ma1/index.cfm   and the javascript used to do the conversions from study statistics to a common metric http://www.lyonsmorris.com/ma1/js/ESConversions.js  hth larry  

---

