<p>This type of data (repeated measures on a group of subjects) lends itself perfectly for a <a href="https://stats.idre.ucla.edu/other/mult-pkg/introduction-to-linear-mixed-models/" rel="nofollow noreferrer"><strong>linear mixed model (LMM)</strong></a> analysis. I'm into psychophysics myself and I have mostly abandoned simple ANOVAs and switched to LMMs almost completely, especially because it can handle missing data, and because LMMs allow for inclusion of trial and session number to correct for them in an elegant manner. In effect, you can potentially take learning effects and/or mental fatigue into account. Between-subject (baseline) differences and learning effects are two of the most pesky problems in psychophysics and repeated measure designs, respectively. This makes the use of LMMs so rewarding in this field.</p>
<p>An LMM has several huge advantages over standard ANOVA analysis and it is expected to replace standard ANOVA analysis in time.</p>
<p>Advantages of LMM include:</p>
<ul>
<li>Subjects (and other variables) can be included as random effect, such that the model corrects for within-subject differences (akin to repeated measures);</li>
<li>Missing data are allowed and corrected for;</li>
<li>Fixed and covariate factors can be included and tested for significance. In your case trial # can be included as a covariate. The LMM will calculate the effect of trial # on your outcome measure and will calculate whether that effect is significant or not (check 'model estimates' under statistics when you use SPSS);</li>
<li>LMMs can be statistically powerful, despite low subject numbers.</li>
</ul>
<p>If you wish to stick to standard RM-ANOVA, you can also opt to <a href="http://www.stat.columbia.edu/%7Egelman/arm/missing.pdf" rel="nofollow noreferrer"><strong>impute your data</strong></a> to correct for missing values. Especially multiple imputation is a robust way of estimating missing data <a href="https://pubmed.ncbi.nlm.nih.gov/27556528/" rel="nofollow noreferrer">(Van Netten <em>et al</em>., 2016)</a>, also available in SPSS.</p>
<p><sub><strong>Reference</strong><br />
<strong>-</strong> <a href="https://pubmed.ncbi.nlm.nih.gov/27556528/" rel="nofollow noreferrer">Van Netten <em>et al</em>. <em>Ear Hear</em> (2017); <strong>38</strong>: 1–6</a></sub></p>

CC BY-SA 4.0
2021-07-19T09:53:07.710Z
Thanks, @AliceD. While I was able to find another way around it, your suggestion seems like a potentially useful solution to similar problems.
zoozoo
CC BY-SA 4.0
2021-07-19T10:32:51.183Z
@zoozoo - feel free to add an answer to your own question and enlighten us with your solution!
AliceD
CC BY-SA 4.0
2021-08-02T17:29:48.047Z
In my case, I was able to avoid the need for such analysis by 're-designing' the study/experiment.
zoozoo
CC BY-SA 4.0
2021-07-12T07:12:00.517Z
2021-07-21T07:05:02.380Z
2021-07-21T07:05:02.380Z
AliceD
<p>Mixed effects regression is somewhat related to repeated measures ANOVA and can handle missing data. However, they assume the missing data are missing at random.</p>

<p>Depending on what type of rejection criteria you are using, that assumption may be broken, in which case I think you're pretty much out of luck without a method to account for that. It may be that it is a sufficiently reasonable assumption, however, even if not strictly warranted.</p>

CC BY-SA 4.0
2020-06-16T14:17:06.610Z
2020-06-16T14:17:06.610Z
Bryan Krause
<p>LMM is a good suggestion, second that.</p>
<p>Consider also fitting a log-normal to the RT's as recommended by Haines &amp; friends
<a href="https://psyarxiv.com/xr7y3/download/?format=pdf" rel="nofollow noreferrer">https://psyarxiv.com/xr7y3/download/?format=pdf</a></p>
<p>You will have more or less uncertainty depending on how many trials you got, but it won't matter that people did different numbers of trials. I find the arguments in this paper for 'one stage' inference (as opposed to two-stage summary) very compelling, see what you think, your mileage may vary. One reason to like it is that it goes very naturally with hierarchical approaches, so it's a natural ally of that LMM suggestion.</p>
<p>It seems likely that the dropped data points are super-fast or super-long RT's that are from some data generating process other than the one you're studying, right? The participant sneezed or snoozed briefly and then went back to doing the task? You can wrap that understanding into the model! If your actual understanding of the data generating process suggests that RT's are usually log normal with a mean that depends on condition, but there are some uniformly distributed contaminants, you can write down a model that says exactly that and do inference with it. There's a nice rundown of the theory/motivation here: <a href="https://www.sciencedirect.com/science/article/pii/S0022249610000593" rel="nofollow noreferrer">https://www.sciencedirect.com/science/article/pii/S0022249610000593</a>
but it's gotten much easier to do since this article was written. There's a blow-by-blow walkthrough with code in this incredibly awesome blog post
<a href="https://www.martinmodrak.cz/2021/04/01/using-brms-to-model-reaction-times-contaminated-with-errors/" rel="nofollow noreferrer">https://www.martinmodrak.cz/2021/04/01/using-brms-to-model-reaction-times-contaminated-with-errors/</a></p>

CC BY-SA 4.0
2021-07-23T09:00:37.887Z
2021-07-23T20:38:30.237Z
2021-07-23T20:38:30.237Z
steveLangsford
<p>I am running a behavioral experiment where a number of subjects (=20 in my case) perform a simple cognitive task. The experiment consists of a fixed number of trials (say, 40 in my case). During each trial, the participant executes a single key-press and the response time (RT) is recorded.</p>

<p>So my recorded data looks like this:</p>

<pre><code>subject-1   -&gt; [Trial-1 RT, Trial-2 RT ... Trial-40 RT]              # (40 trials)
subject-2   -&gt; [Trial-1 RT, Trial-2 RT ... Trial-40 RT]              # (40 trials)
...        ...       ...
subject-20  -&gt; [Trial-1 RT, Trial-2 RT ... Trial-40 RT]              # (40 trials)
</code></pre>

<p>Now based on some RT criterion, a few trials are removed for each subject. This resulting data looks like this:</p>

<pre><code>subject-1   -&gt; [Trial-1 RT, Trial-3 RT ... Trial-40 RT]              # (32 trials)
subject-2   -&gt; [Trial-1 RT, Trial-2 RT ... Trial-38 RT]              # (36 trials)
...        ...       ...
subject-20  -&gt; [Trial-3 RT, Trial-8 RT ... Trial-40 RT]              # (28 trials)
</code></pre>

<p>The removal of a few trials result in a non-uniform number of data-points for each subject. Eg. subject-1, subject-2 and subject-20 have 32, 36 and 28 trials respectively.</p>

<p>Now I want to test the effect of trials on response times, what statistical methods should I use?</p>

<p>I know that when I am <strong>not</strong> removing the data, I have a nice 20*40 (subjects * trials) data matrix on which I can perform repeated measures ANOVA (within-subject) to see the effect of trials on response times. But how should I go about it if I am removing a couple of trials?</p>

<methodology><experimental-psychology><statistics><psychophysics>
What statistical tests should be performed to test effect of trials on response times where each subject has different number of data points?
zoozoo
# What statistical tests should be performed to test effect of trials on response times where each subject has different number of data points?

(Asked by: username_0 on 2020-06-16)

I am running a behavioral experiment where a number of subjects (=20 in my case) perform a simple cognitive task. The experiment consists of a fixed number of trials (say, 40 in my case). During each trial, the participant executes a single key-press and the response time (RT) is recorded.
So my recorded data looks like this:
subject-1   -> [Trial-1 RT, Trial-2 RT ... Trial-40 RT]              # (40 trials)
subject-2   -> [Trial-1 RT, Trial-2 RT ... Trial-40 RT]              # (40 trials)
...        ...       ...
subject-20  -> [Trial-1 RT, Trial-2 RT ... Trial-40 RT]              # (40 trials)

Now based on some RT criterion, a few trials are removed for each subject. This resulting data looks like this:
subject-1   -> [Trial-1 RT, Trial-3 RT ... Trial-40 RT]              # (32 trials)
subject-2   -> [Trial-1 RT, Trial-2 RT ... Trial-38 RT]              # (36 trials)
...        ...       ...
subject-20  -> [Trial-3 RT, Trial-8 RT ... Trial-40 RT]              # (28 trials)

The removal of a few trials result in a non-uniform number of data-points for each subject. Eg. subject-1, subject-2 and subject-20 have 32, 36 and 28 trials respectively.
Now I want to test the effect of trials on response times, what statistical methods should I use?
I know that when I am not removing the data, I have a nice 20*40 (subjects * trials) data matrix on which I can perform repeated measures ANOVA (within-subject) to see the effect of trials on response times. But how should I go about it if I am removing a couple of trials?

**Tags:** methodology, experimental-psychology, statistics, psychophysics


---

## 3 answers

### Answer by username_1 on 2021-07-12. Score: 3

This type of data (repeated measures on a group of subjects) lends itself perfectly for a [**linear mixed model (LMM)**](https://stats.idre.ucla.edu/other/mult-pkg/introduction-to-linear-mixed-models/) analysis. I'm into psychophysics myself and I have mostly abandoned simple ANOVAs and switched to LMMs almost completely, especially because it can handle missing data, and because LMMs allow for inclusion of trial and session number to correct for them in an elegant manner. In effect, you can potentially take learning effects and/or mental fatigue into account. Between-subject (baseline) differences and learning effects are two of the most pesky problems in psychophysics and repeated measure designs, respectively. This makes the use of LMMs so rewarding in this field.

An LMM has several huge advantages over standard ANOVA analysis and it is expected to replace standard ANOVA analysis in time.

Advantages of LMM include:

* Subjects (and other variables) can be included as random effect, such that the model corrects for within-subject differences (akin to repeated measures);
* Missing data are allowed and corrected for;
* Fixed and covariate factors can be included and tested for significance. In your case trial # can be included as a covariate. The LMM will calculate the effect of trial # on your outcome measure and will calculate whether that effect is significant or not (check 'model estimates' under statistics when you use SPSS);
* LMMs can be statistically powerful, despite low subject numbers.

If you wish to stick to standard RM-ANOVA, you can also opt to [**impute your data**](http://www.stat.columbia.edu/%7Egelman/arm/missing.pdf) to correct for missing values. Especially multiple imputation is a robust way of estimating missing data [(Van Netten *et al*., 2016)](https://pubmed.ncbi.nlm.nih.gov/27556528/), also available in SPSS.

**Reference**  
**-** [Van Netten *et al*. *Ear Hear* (2017); **38**: 1–6](https://pubmed.ncbi.nlm.nih.gov/27556528/)

> **Comments:**
> **username_0**: Thanks, @username_1. While I was able to find another way around it, your suggestion seems like a potentially useful solution to similar problems.  
> **username_1**: @username_0 - feel free to add an answer to your own question and enlighten us with your solution!  
> **username_0**: In my case, I was able to avoid the need for such analysis by 're-designing' the study/experiment.  

---

### Answer by username_2 on 2020-06-16. Score: 1

Mixed effects regression is somewhat related to repeated measures ANOVA and can handle missing data. However, they assume the missing data are missing at random.

Depending on what type of rejection criteria you are using, that assumption may be broken, in which case I think you're pretty much out of luck without a method to account for that. It may be that it is a sufficiently reasonable assumption, however, even if not strictly warranted.


---

### Answer by username_3 on 2021-07-23. Score: 1

LMM is a good suggestion, second that.

Consider also fitting a log-normal to the RT's as recommended by Haines & friends
<https://psyarxiv.com/xr7y3/download/?format=pdf>

You will have more or less uncertainty depending on how many trials you got, but it won't matter that people did different numbers of trials. I find the arguments in this paper for 'one stage' inference (as opposed to two-stage summary) very compelling, see what you think, your mileage may vary. One reason to like it is that it goes very naturally with hierarchical approaches, so it's a natural ally of that LMM suggestion.

It seems likely that the dropped data points are super-fast or super-long RT's that are from some data generating process other than the one you're studying, right? The participant sneezed or snoozed briefly and then went back to doing the task? You can wrap that understanding into the model! If your actual understanding of the data generating process suggests that RT's are usually log normal with a mean that depends on condition, but there are some uniformly distributed contaminants, you can write down a model that says exactly that and do inference with it. There's a nice rundown of the theory/motivation here: <https://www.sciencedirect.com/science/article/pii/S0022249610000593>
but it's gotten much easier to do since this article was written. There's a blow-by-blow walkthrough with code in this incredibly awesome blog post
<https://www.martinmodrak.cz/2021/04/01/using-brms-to-model-reaction-times-contaminated-with-errors/>


---

