<p>A power analysis is typically based on the difference in the mean to be detected and the expected standard deviation of the data. The amount of variation in the data (i.e., the size of the standard deviation) typically depends on individual differences in the subjects and measurement noise. The measurement noise depends on a number of things, but in many experiments can be reduced by running more trials. In the minimal risk experiments I am most familiar with, we run so many trials that we reduce the trial related noise to inconsequential amounts. We tend to use wildly inefficient methods for collecting the trials also (our stimuli and response interval are too long, we use multiple intervals, and our adaptive procedures are not optimized).</p>

<p>In order to minimize the data collection/maximize power (and this should not be the only goal) the stopping rule generally cannot be a number of trials, but rather a precision criteria. There are procedures (e.g., <a href="http://www.sciencedirect.com/science/article/pii/S0042698902003966" rel="nofollow noreferrer">ZEST</a>) which can be set up to use a fixed number of trials or a target precision.</p>

CC BY-SA 3.0
2017-03-30T19:27:28.820Z
2017-03-30T19:27:28.820Z
StrongBad
<p>I'm currently designing an experiment with several conditions - how do I decide how many repetitions / stimuli within each condition?</p>

<p>I know that I can justify sample size with an a priori power analysis...is there any common way to choose how many trials within each of my conditions?</p>

<p>I was using sample size/power as an example of something where there is a particular approach to help decide.....what I am actually asking about is the number of <strong>trials</strong> i.e. if I have three conditions in my experiment, how many stimuli should I design for each of the conditions?</p>

CC BY-SA 3.0
2017-03-27T16:05:36.267Z
If you don't want do any calculations you could look at similar researches and see how many trials they have used. For ways of calculating effect size/power or the like, you may want to check [stats.se], the statistics SE
Robin Kramer-ten Have
CC BY-SA 3.0
2017-03-29T07:07:28.717Z
Related: http://cogsci.stackexchange.com/q/3384/11318
Robin Kramer-ten Have
CC BY-SA 3.0
2017-03-29T15:42:50.163Z
Obviously, the choice of sample size is a compromise between the results of the power analysis (larger sample size = higher power) and the costs of the larger sample (larger sample = higher cost). It's not clear, what kind of answer are You @LennaKB looking for? Are you looking for a formal decision model that will tell You what sample size to choose given a specific cost function? Does this question ask what a power analysis would suggest? Are there some aspects of the current scenario (eg the hierarchical structure of the exp.) that make standard solutions from power analysis not applicable?
matus
CC BY-SA 3.0
2017-03-30T17:01:56.520Z
I am asking about trials rather than sample size (just used sample size as an example of something that you can use a certain approach to decide).
LennaKB
<cognitive-psychology><experimental-psychology><experiment-design>
How do you decide how many trials to have in an experiment?
LennaKB
# How do you decide how many trials to have in an experiment?

(Asked by: username_0 on 2017-03-27)

I'm currently designing an experiment with several conditions - how do I decide how many repetitions / stimuli within each condition?

I know that I can justify sample size with an a priori power analysis...is there any common way to choose how many trials within each of my conditions?

I was using sample size/power as an example of something where there is a particular approach to help decide.....what I am actually asking about is the number of **trials** i.e. if I have three conditions in my experiment, how many stimuli should I design for each of the conditions?

**Tags:** cognitive-psychology, experimental-psychology, experiment-design

> **Comments:**
> **username_1**: If you don't want do any calculations you could look at similar researches and see how many trials they have used. For ways of calculating effect size/power or the like, you may want to check [stats.se], the statistics SE  
> **username_1**: Related: http://cogsci.stackexchange.com/q/3384/11318  
> **username_2**: Obviously, the choice of sample size is a compromise between the results of the power analysis (larger sample size = higher power) and the costs of the larger sample (larger sample = higher cost). It's not clear, what kind of answer are You @username_0 looking for? Are you looking for a formal decision model that will tell You what sample size to choose given a specific cost function? Does this question ask what a power analysis would suggest? Are there some aspects of the current scenario (eg the hierarchical structure of the exp.) that make standard solutions from power analysis not applicable?  
> **username_0**: I am asking about trials rather than sample size (just used sample size as an example of something that you can use a certain approach to decide).  

---

## 1 answers

### Answer by username_3 on 2017-03-30. Score: 1

A power analysis is typically based on the difference in the mean to be detected and the expected standard deviation of the data. The amount of variation in the data (i.e., the size of the standard deviation) typically depends on individual differences in the subjects and measurement noise. The measurement noise depends on a number of things, but in many experiments can be reduced by running more trials. In the minimal risk experiments I am most familiar with, we run so many trials that we reduce the trial related noise to inconsequential amounts. We tend to use wildly inefficient methods for collecting the trials also (our stimuli and response interval are too long, we use multiple intervals, and our adaptive procedures are not optimized).

In order to minimize the data collection/maximize power (and this should not be the only goal) the stopping rule generally cannot be a number of trials, but rather a precision criteria. There are procedures (e.g., [ZEST](http://www.sciencedirect.com/science/article/pii/S0042698902003966)) which can be set up to use a fixed number of trials or a target precision.


---

