<p>Could you maybe check in simulation?
I'm a huge fan of the workflow described here:
<a href="https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html" rel="nofollow noreferrer">https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html</a></p>

<p>You are asking question #3 of 4, the "inferential adequacy" one, right? Can you run some simulations with varying 'true stimulus-set gap' and just see how big a gap you can tolerate with your design?</p>

<p>You don't need to know the true data generating process to do this, generating responses from the model tells you something useful about the model+design setup, in this workflow reality has to wait until question #4. Also an important question, but checking against reality comes <em>after</em> passing the tests you're talking about.</p>

<p>Although you don't need to know how people actually come up with their responses, you <em>do</em> need to be Bayesian enough to be comfortable putting priors on all the parameters. If it's important to convince a colleague with the results, might be worth checking that's not a dealbreaker for them!</p>

CC BY-SA 4.0
2020-05-29T13:40:50.070Z
2020-05-29T13:40:50.070Z
steveLangsford
<p>I'm designing a psychology experiment in which we're interested in how working memory load affects performance in a theory-of-mind task. Independent variable is working memory load, dependent variable is theory-of-mind task performance.</p>

<p>In our within-subjects design, we have 2 working memory conditions (high load and low load). We have 2 stimulus sets (one for each working memory condition, counterbalanced across participants) to prevent carry-over effects between working memory conditions.</p>

<p>The problem is that performance in our task is driven not only by working memory condition, but also by stimulus set to a large extent. My colleague believes this can be dealt with using mixed models. However, I have some doubts about taking this approach, and would rather take the time to adjust the stimulus sets so that performance within each subject is as comparable as possible between both stimulus sets (under the same working memory condition).</p>

<p>If I were to take my colleague's advice, would there be any issues for example with power? And are there resources that you could point me to so that I can understand this issue better?</p>

<p>Edit: There were some questions about sample size and analysis setup. I'm aiming for around 40 participants, but have only just started collecting data. </p>

<p>To clarify, I'm not against using mixed models per se, but I'm wondering: 1) whether there might be a loss in power which is proportionate to the imbalance in effects between the two stimulus types, 2) whether I may need to take steps to compensate for this loss in power, and 3) if so, what steps to take (e.g., is there a way to calculate, roughly, how many more participants I would need given a certain effect size for stimulus type? Or is there perhaps a smarter way to set up the analyses?). </p>

<p>The analysis setup is to perform a linear mixed model, with dependent variable as proportion of correct responses, with fixed effects as working memory load (high vs. low), stimulus type (set 1 or 2), and their interaction, and with mixed effects as intercepts for subjects and items. Then we'd compare this fit with fit of a reduced model without these fixed effects. </p>

<p>I hope this clarifies my question, and appreciate help answering the 3 questions stated in my edit.</p>

CC BY-SA 4.0
2020-05-27T13:57:32.397Z
This sounds similar to something I struggled with many years ago, and have never fully resolved: https://psychology.stackexchange.com/q/3722/21 I ended up making up my own stimulus set, prioritizing ecological validity, at the cost of power. https://www.sciencedirect.com/science/article/pii/S0747563216302308?casa_token=2H94jSOrZ-AAAAAA:b9WEt4DAzPVQ8k0SOeZZ6mK9FAG62q1TkUsmo4JMd-orn-HlbB3BcfbAaJfqw0qoFGJDfRbDHYk There is a tradeoff between power and ecological validity of the stimulus set, if you ask me.
Steven Jeuris
CC BY-SA 4.0
2020-05-27T14:35:17.853Z
You would have to provide more information on how your analyses are set up, but I don't see how you can avoid a repeated measures statistical analysis here without committing some statistical faux pas; mixed models are one flexible way to do that. You don't gain real power by using a flawed statistical design, just fake power at the expense of inflated type I errors.
Bryan Krause
CC BY-SA 4.0
2020-05-27T15:14:50.423Z
I agree with Bryan, more information is needed to answer this Q. Number of participants, other variables that can be thrown into the LMM etc etc
AliceD
CC BY-SA 4.0
2020-05-28T18:16:42.373Z
I've now edited my post to answer questions about sample size and analysis setup.
Lionel Newman
<statistics><experiment-design>
Power issues with mixed models
Lionel Newman
# Power issues with mixed models

(Asked by: username_0 on 2020-05-27)

I'm designing a psychology experiment in which we're interested in how working memory load affects performance in a theory-of-mind task. Independent variable is working memory load, dependent variable is theory-of-mind task performance.

In our within-subjects design, we have 2 working memory conditions (high load and low load). We have 2 stimulus sets (one for each working memory condition, counterbalanced across participants) to prevent carry-over effects between working memory conditions.

The problem is that performance in our task is driven not only by working memory condition, but also by stimulus set to a large extent. My colleague believes this can be dealt with using mixed models. However, I have some doubts about taking this approach, and would rather take the time to adjust the stimulus sets so that performance within each subject is as comparable as possible between both stimulus sets (under the same working memory condition).

If I were to take my colleague's advice, would there be any issues for example with power? And are there resources that you could point me to so that I can understand this issue better?

Edit: There were some questions about sample size and analysis setup. I'm aiming for around 40 participants, but have only just started collecting data.

To clarify, I'm not against using mixed models per se, but I'm wondering: 1) whether there might be a loss in power which is proportionate to the imbalance in effects between the two stimulus types, 2) whether I may need to take steps to compensate for this loss in power, and 3) if so, what steps to take (e.g., is there a way to calculate, roughly, how many more participants I would need given a certain effect size for stimulus type? Or is there perhaps a smarter way to set up the analyses?).

The analysis setup is to perform a linear mixed model, with dependent variable as proportion of correct responses, with fixed effects as working memory load (high vs. low), stimulus type (set 1 or 2), and their interaction, and with mixed effects as intercepts for subjects and items. Then we'd compare this fit with fit of a reduced model without these fixed effects.

I hope this clarifies my question, and appreciate help answering the 3 questions stated in my edit.

**Tags:** statistics, experiment-design

> **Comments:**
> **username_1**: This sounds similar to something I struggled with many years ago, and have never fully resolved: https://psychology.stackexchange.com/q/3722/21 I ended up making up my own stimulus set, prioritizing ecological validity, at the cost of power. https://www.sciencedirect.com/science/article/pii/S0747563216302308?casa_token=2H94jSOrZ-AAAAAA:b9WEt4DAzPVQ8k0SOeZZ6mK9FAG62q1TkUsmo4JMd-orn-HlbB3BcfbAaJfqw0qoFGJDfRbDHYk There is a tradeoff between power and ecological validity of the stimulus set, if you ask me.  
> **username_2**: You would have to provide more information on how your analyses are set up, but I don't see how you can avoid a repeated measures statistical analysis here without committing some statistical faux pas; mixed models are one flexible way to do that. You don't gain real power by using a flawed statistical design, just fake power at the expense of inflated type I errors.  
> **username_3**: I agree with Bryan, more information is needed to answer this Q. Number of participants, other variables that can be thrown into the LMM etc etc  
> **username_0**: I've now edited my post to answer questions about sample size and analysis setup.  

---

## 1 answers

### Answer by username_4 on 2020-05-29. Score: 2

Could you maybe check in simulation?
I'm a huge fan of the workflow described here:
<https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html>

You are asking question #3 of 4, the "inferential adequacy" one, right? Can you run some simulations with varying 'true stimulus-set gap' and just see how big a gap you can tolerate with your design?

You don't need to know the true data generating process to do this, generating responses from the model tells you something useful about the model+design setup, in this workflow reality has to wait until question #4. Also an important question, but checking against reality comes *after* passing the tests you're talking about.

Although you don't need to know how people actually come up with their responses, you *do* need to be Bayesian enough to be comfortable putting priors on all the parameters. If it's important to convince a colleague with the results, might be worth checking that's not a dealbreaker for them!


---

