<p>The purpose of reviewing the literature for effect sizes is to form an estimate of what effect size you might expect in your present study. </p>

<p><strong>Existing meta-analysis:</strong> The principles and techniques of meta-analysis provide a good starting point for generating a predicted effect size. 
If a meta-analysis has already been conducted, then the estimated population mean and confidence intervals of effect sizes is a useful piece of information. 
In some cases there will also be a <strong>moderator</strong> analysis reported which tests whether the effect size varies as a function of the moderator (e.g., adults versus children; lab versus field, etc.). If your study matches one level of an influential moderator, then it may be worth weighting estimates more by the estimated effect size for that moderator.</p>

<p><strong>No existing meta-analysis:</strong> If no meta-analysis exists or if one exists but is out of date, then you may want to perform your own meta-analysis. There are several great <a href="http://www.meta-analysis.com/pages/books.html" rel="nofollow noreferrer">books on meta-analysis</a>. I've personally found Borenstein et al (2011) to be a good introduction and Hunter and Schmidt (2004) to be useful  particularly if you are interested in correlations and various corrections. Here is a <a href="http://omerad.msu.edu/ebm/Meta-analysis/Meta3.html" rel="nofollow noreferrer">link to an overview of meta-analysis</a>.
Conducting a rigorous meta-analysis for publication is a technical and resource intensive process. However, if all you want is a ball-park estimate of expected effect size for sample size and study design considerations (i.e., your not doing an explicit meta-analysis publication), then it may be more pragmatic to just do a quick informal meta-analysis.  In some cases there might only be one or a few studies which are similar to your study. In this case, the meta-analysis can be very simple.</p>

<p><strong>Generalisation:</strong> Another issue that comes up is generalisation. Students sometimes say to me that there have been no other studies like the one being conducted. This may be true, but at some level of generality this is almost always false. If a study is about a particular training program in the work place, then a meta-analysis of training in general may be informative. If the intervention has only been used with teenagers and is now being applied to adults, the meta-analysis of teenagers would be relevant. Broadly, theory should be applied to assess the likely generality of any broader research to the present study. </p>

<p><strong>Theory:</strong> Theory is important in many respects. Theory may suggest that a particular modification to the design should increase or decrease the effect size relative to a previous studies.</p>

<p><strong>Aggregating estimates:</strong> In general it is is important to consider how different effect size estimates should be aggregated. The more similar a previous study is to the present study, the more weight should be given to any effect size estimate. In this case, similarity requires theoretical judgement about what are the important factors that might moderate the effect. Similarly any aggregation needs to be mindful of sample size considerations where larger sample sizes will increase the precision of the estimate in any given study. Thus, at the ultra-specific, the present study may be an exact replication of a previous study. At another level, there may be meta-analytic evidence from similar studies. At an even broader level Jacob Cohen reluctantly proposed various rough rules of thumb of what to expect for small, medium, and large effects in behavioural science studies.</p>

<p><strong>Importance of uncertainty:</strong> Another point is that you don't actually know the population effect size for your study. That's why you are doing the study. There will always be uncertainty about your effect size estimate. It can even be useful to quantify that uncertainty either using meta-analytic confidence intervals or something more ad hoc. 
This is analogous to the concept of a subjective bayesian prior probability on a parameter of interest. See for example, this discussion of <a href="https://stats.stackexchange.com/questions/1/eliciting-priors-from-experts">eliciting priors from experts</a>. Meta-analysis provides some methods for quantifying uncertainty in an effect size estimate, but this may not be entirely appropriate given that you are not trying to generalise to the variation in the whole world of effect sizes. Rather you are trying to generalise a prediction for a specific study, and you may be pooling many different pieces of information, some quantitative, and some qualitative.</p>

<hr>

<p><strong>By way of example</strong>, I do research on personality faking. I am aware of a meta-analysis of the mean effect size of faking versus honest personality responses (e.g., perhaps around d = .6). However, I also know from previous research that the type of faking instructions makes a big difference to the expected effect size (e.g., if you tell people to fake you often get approximate d = 1.0 and if you just put people in a context with a motivation to fake you get approximate d = .3). Thus, If I were calculating expected effect size for any given study I would combine knowledge of the nature of my faking instructions with various meta-analytic estimates. I'd weight my own previous studies highly because they typically share more similar features (e.g., same personality test, similar participants, etc.) than the general literature. I'd also acknowledge the uncertainty in my estimate.</p>

<h3>References</h3>

<ul>
<li>Borenstein, M., Hedges, L. V., Higgins, J. P., &amp; Rothstein, H. R. (2011). Introduction to meta-analysis. Wiley.</li>
<li>Hunter, J. E., &amp; Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. SAGE Publications, Incorporated.</li>
</ul>

CC BY-SA 3.0
2013-04-05T05:40:27.830Z
2013-04-05T05:45:54.917Z
2017-04-13T12:44:18.443Z
Jeromy Anglim
<p>If you're trying to work out a standardized effect size in order to calculate power for your study then it doesn't matter whether any studies like yours have been done. It's important to find studies using the same dependent measure so that you have an estimate of variability but evidence of the size of prior effects is less important.  If you have the variance, and a bit of theory, then you can estimate the standardized effect size for the minimum effect that would be meaningful for your study.  Instead of asking how big an effect others have found, you should ask yourself how big an effect matters.  If it's smaller than the literature and you aim your design at the literature you'll tend to miss effects that would have been important to you.  When you think about it that's kind of a waste of all of the subjects you did run in your study.</p>

<p>Estimating that effect size only requires a reasonable theory and sampling of the literature, and some research that can give you variability estimates for the effect</p>

<p>As an alternate, you could consider estimating the minimum LSD you want for the study in advance, and that can just be in the units of the dependent variable.  For example, reaction time effects in a particular test might only be meaningful over 7ms and you can make that your planned LSD.  Anything less isn't expected to be meaningful. Unlike your hypothesis test, you are allowed to then just keep testing as you go along until your data achieve this LSD.  It will not affect alpha or power.  Keep in mind, the only criteria you can use for adding subjects is the preplanned LSD, not whether you have a significant effect or not.</p>

<p>This has a number of benefits but the most important one is that it insures that your study can say something whether you find a significant effect or not.  Having studies that can speak to both sides of an issue makes them much more valuable.</p>

<p>Finally, I think the difficulty of a meta-analysis is exaggerated.  All research is hard and meta-analyses aren't really any more difficult.  Sometimes they're easier.  But more importantly, if you can't find one for the topic at hand then conducting one has great benefits.  It will instantly give you an important paper in the field, expert status, and greatly inform your future research.  You've got to read the papers anyway.  Make notes about the results and work out their effects as you do it.  Then do the meta-analysis after.  You just need to extract the effect size and N.  The hard part of meta is proper interpretation, not conducting it (in fact, <a href="http://www.metafor-project.org/doku.php" rel="nofollow">metafor in R</a> makes it a bit too easy).  What's paradoxical is that this is MORE important for you to do if you're not statistically astute than if you are.  That's because someone who just gets statistics quite well is more likely to get an accurate impression of the literature without having to do all of the meta-analysis calculations.</p>

CC BY-SA 3.0
2013-04-05T12:31:35.690Z
2013-04-15T01:29:25.260Z
2013-04-15T01:29:25.260Z
John
<p>When planning a psychological study, researchers are often asked to formulate an estimate of  expected effect sizes for important hypotheses in the study. This might be the expected group mean difference (e.g., Cohen's d), the correlation between two variables, the odds ratio, or any number of other effects of interest.
Such estimates can then link into power analysis and sample size calculations.</p>

<p>However, often formulating such an estimate is a difficult task. The exact study has not been performed before. Meta-analyses may only be partially related to the hypothesis of interest. Various modifications to the design might be theorised to change the effect size from previous studies. This leads to the question of how to systematically integrate such information to formulate an expected effect.</p>

<ul>
<li>What is a good strategy for formulating an expected effect size in a psychological study, particularly where the study has not been performed before?</li>
<li>Are there any scientific articles that attempt to elucidate best practice in effect size estimation in a psychological research context?</li>
</ul>

<p><em>Note: A student recently asked me this question, thus I thought I'd post it here to draw on the expertise of others. I will post my answer below as well, but hopefully other people add answers also.</em></p>

<methodology><statistics>
How to estimate an expected effect size for a planned psychology study?
Jeromy Anglim
# How to estimate an expected effect size for a planned psychology study?

(Asked by: username_0 on 2013-04-05)

When planning a psychological study, researchers are often asked to formulate an estimate of expected effect sizes for important hypotheses in the study. This might be the expected group mean difference (e.g., Cohen's d), the correlation between two variables, the odds ratio, or any number of other effects of interest.
Such estimates can then link into power analysis and sample size calculations.

However, often formulating such an estimate is a difficult task. The exact study has not been performed before. Meta-analyses may only be partially related to the hypothesis of interest. Various modifications to the design might be theorised to change the effect size from previous studies. This leads to the question of how to systematically integrate such information to formulate an expected effect.

* What is a good strategy for formulating an expected effect size in a psychological study, particularly where the study has not been performed before?
* Are there any scientific articles that attempt to elucidate best practice in effect size estimation in a psychological research context?

*Note: A student recently asked me this question, thus I thought I'd post it here to draw on the expertise of others. I will post my answer below as well, but hopefully other people add answers also.*

**Tags:** methodology, statistics


---

## 2 answers

### Answer by username_0 on 2013-04-05. Score: 6

The purpose of reviewing the literature for effect sizes is to form an estimate of what effect size you might expect in your present study.

**Existing meta-analysis:** The principles and techniques of meta-analysis provide a good starting point for generating a predicted effect size.
If a meta-analysis has already been conducted, then the estimated population mean and confidence intervals of effect sizes is a useful piece of information.
In some cases there will also be a **moderator** analysis reported which tests whether the effect size varies as a function of the moderator (e.g., adults versus children; lab versus field, etc.). If your study matches one level of an influential moderator, then it may be worth weighting estimates more by the estimated effect size for that moderator.

**No existing meta-analysis:** If no meta-analysis exists or if one exists but is out of date, then you may want to perform your own meta-analysis. There are several great [books on meta-analysis](http://www.meta-analysis.com/pages/books.html). I've personally found Borenstein et al (2011) to be a good introduction and Hunter and Schmidt (2004) to be useful particularly if you are interested in correlations and various corrections. Here is a [link to an overview of meta-analysis](http://omerad.msu.edu/ebm/Meta-analysis/Meta3.html).
Conducting a rigorous meta-analysis for publication is a technical and resource intensive process. However, if all you want is a ball-park estimate of expected effect size for sample size and study design considerations (i.e., your not doing an explicit meta-analysis publication), then it may be more pragmatic to just do a quick informal meta-analysis. In some cases there might only be one or a few studies which are similar to your study. In this case, the meta-analysis can be very simple.

**Generalisation:** Another issue that comes up is generalisation. Students sometimes say to me that there have been no other studies like the one being conducted. This may be true, but at some level of generality this is almost always false. If a study is about a particular training program in the work place, then a meta-analysis of training in general may be informative. If the intervention has only been used with teenagers and is now being applied to adults, the meta-analysis of teenagers would be relevant. Broadly, theory should be applied to assess the likely generality of any broader research to the present study.

**Theory:** Theory is important in many respects. Theory may suggest that a particular modification to the design should increase or decrease the effect size relative to a previous studies.

**Aggregating estimates:** In general it is is important to consider how different effect size estimates should be aggregated. The more similar a previous study is to the present study, the more weight should be given to any effect size estimate. In this case, similarity requires theoretical judgement about what are the important factors that might moderate the effect. Similarly any aggregation needs to be mindful of sample size considerations where larger sample sizes will increase the precision of the estimate in any given study. Thus, at the ultra-specific, the present study may be an exact replication of a previous study. At another level, there may be meta-analytic evidence from similar studies. At an even broader level Jacob Cohen reluctantly proposed various rough rules of thumb of what to expect for small, medium, and large effects in behavioural science studies.

**Importance of uncertainty:** Another point is that you don't actually know the population effect size for your study. That's why you are doing the study. There will always be uncertainty about your effect size estimate. It can even be useful to quantify that uncertainty either using meta-analytic confidence intervals or something more ad hoc.
This is analogous to the concept of a subjective bayesian prior probability on a parameter of interest. See for example, this discussion of [eliciting priors from experts](https://stats.stackexchange.com/questions/1/eliciting-priors-from-experts). Meta-analysis provides some methods for quantifying uncertainty in an effect size estimate, but this may not be entirely appropriate given that you are not trying to generalise to the variation in the whole world of effect sizes. Rather you are trying to generalise a prediction for a specific study, and you may be pooling many different pieces of information, some quantitative, and some qualitative.

---

**By way of example**, I do research on personality faking. I am aware of a meta-analysis of the mean effect size of faking versus honest personality responses (e.g., perhaps around d = .6). However, I also know from previous research that the type of faking instructions makes a big difference to the expected effect size (e.g., if you tell people to fake you often get approximate d = 1.0 and if you just put people in a context with a motivation to fake you get approximate d = .3). Thus, If I were calculating expected effect size for any given study I would combine knowledge of the nature of my faking instructions with various meta-analytic estimates. I'd weight my own previous studies highly because they typically share more similar features (e.g., same personality test, similar participants, etc.) than the general literature. I'd also acknowledge the uncertainty in my estimate.

### References

* Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2011). Introduction to meta-analysis. Wiley.
* Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. SAGE Publications, Incorporated.


---

### Answer by username_1 on 2013-04-05. Score: 5

If you're trying to work out a standardized effect size in order to calculate power for your study then it doesn't matter whether any studies like yours have been done. It's important to find studies using the same dependent measure so that you have an estimate of variability but evidence of the size of prior effects is less important. If you have the variance, and a bit of theory, then you can estimate the standardized effect size for the minimum effect that would be meaningful for your study. Instead of asking how big an effect others have found, you should ask yourself how big an effect matters. If it's smaller than the literature and you aim your design at the literature you'll tend to miss effects that would have been important to you. When you think about it that's kind of a waste of all of the subjects you did run in your study.

Estimating that effect size only requires a reasonable theory and sampling of the literature, and some research that can give you variability estimates for the effect

As an alternate, you could consider estimating the minimum LSD you want for the study in advance, and that can just be in the units of the dependent variable. For example, reaction time effects in a particular test might only be meaningful over 7ms and you can make that your planned LSD. Anything less isn't expected to be meaningful. Unlike your hypothesis test, you are allowed to then just keep testing as you go along until your data achieve this LSD. It will not affect alpha or power. Keep in mind, the only criteria you can use for adding subjects is the preplanned LSD, not whether you have a significant effect or not.

This has a number of benefits but the most important one is that it insures that your study can say something whether you find a significant effect or not. Having studies that can speak to both sides of an issue makes them much more valuable.

Finally, I think the difficulty of a meta-analysis is exaggerated. All research is hard and meta-analyses aren't really any more difficult. Sometimes they're easier. But more importantly, if you can't find one for the topic at hand then conducting one has great benefits. It will instantly give you an important paper in the field, expert status, and greatly inform your future research. You've got to read the papers anyway. Make notes about the results and work out their effects as you do it. Then do the meta-analysis after. You just need to extract the effect size and N. The hard part of meta is proper interpretation, not conducting it (in fact, [metafor in R](http://www.metafor-project.org/doku.php) makes it a bit too easy). What's paradoxical is that this is MORE important for you to do if you're not statistically astute than if you are. That's because someone who just gets statistics quite well is more likely to get an accurate impression of the literature without having to do all of the meta-analysis calculations.


---

