---
author:
- 'Sascha Meyen$^1$, Iris A. Zerweck$^1$, Catarina Amado$^1$, Ulrike von Luxburg$^2$, and Volker H. Franz$^1$'
title: '**The unconscious priming fallacy: When can scientists truly claim an indirect task advantage?**'
---

Research on consciousness and its cerebral substrates has far-reaching implications and received substantial attention in recent years [@Michel_etal_19]. A driving factor have been reports that masked stimuli that are not consciously perceived can nevertheless affect behavioral responses and brain activity [@Kouider_Dehaene_07; @vanDenBussche_09]. The exciting claim here is that unconscious processing might be more than a mere residue of conscious processing and may be performed by different neuronal processes than conscious processing. Such results impact current theories about the functional role of consciousness [@dehaene2017consciousness; @Kouider_Dehaene_07; @vanDenBussche_09; @Sklar_etal_12; @Hassin_13], might suggest parallel neuronal routes for unconscious vs. conscious processing [@Morris_etal_99], and might support theories of superior unconscious processing [@Custers_Aarts_10; @dijksterhuis2006making; @tenBrinke_etal_16].

Here, we scrutinize one of the most frequently used approaches in this field, which we dub the **standard reasoning of unconscious priming**. We show that this reasoning is flawed for mathematical reasons, fails to provide meaningful interpretation of the data, and needs to be replaced by a more appropriate analysis. Because many studies have used the standard reasoning, a large body of literature needs reassessment. This has the potential to drastically change our views on the relationship of conscious vs. unconscious processing and their neuronal underpinnings. The fallacy affects a wide range of research areas because the standard reasoning has been employed on such diverse topics as, for example, unconscious processing of semantic meaning [@Dehaene_etal_98], motivation [@Pessiglione_etal_07], emotion [@Morris_etal_98], cognitive control [@van2010unconscious], and detection of liars [@tenBrinke_14]. To assess how seriously the literature is affected, we have proceeded in three strands: (a) We replicated the behavioral part of a landmark study [@Dehaene_etal_98] and show that the appropriate analysis of the data does not support unconscious priming (in contrast to the claims of the original study). (b) We developed statistical methods to reanalyze published studies based on the published $t$ and $F$ statistics (because access to the full trial-by-trial data is often lacking). We validated this approach, for example, by showing that our reanalysis of the published data of is consistent with the results of our replication. (c) We used our methods to reanalyze 15 highly influential studies (with a total of [3277]{} citations in Web of Science). Even though all these studies used the standard reasoning to infer unconscious processing, their data tell a different story.

The standard reasoning of unconscious priming {#the-standard-reasoning-of-unconscious-priming .unnumbered}
=============================================

Before describing our newly developed methods and results we need to describe the standard reasoning and why exactly it is so problematic.

Illustrative example: Unconscious lie detection {#illustrative-example-unconscious-lie-detection .unnumbered}
-----------------------------------------------

As an illustrative example, consider the study by who reported that humans can detect liars better unconsciously than consciously. Participants first watched videos of suspects who were either lying or telling the truth. Then participants performed two tasks. In the **direct task**, they viewed pictures of the suspects and judged who had been lying or telling the truth. Participants performed poorly with an accuracy of only 49.62%-correct (with chance level being 50%). In the **indirect task**, participants categorized words, such as “deceitful” or “honest”, into the categories lying or truth–telling. Before each word, a masked picture of a suspect was flashed. In consequence, participants’ reaction times (RTs) were faster when congruent stimuli were presented (e.g., a lying suspect before a lie–related word) as compared to incongruent stimuli. Based on this congruency effect, concluded that the indirect task indicated relatively good accuracy for the discrimination between truth–tellers and liars and that this sensitivity was better than that in the direct task: “indirect measures of deception detection are significantly more accurate than direct measures” (p. 1098, Abstract). Before we proceed, let us stress two things here: (a) Up to this point, attempted to compare only performances in the direct and indirect tasks, independent of considerations of which task is measuring conscious or unconscious processes. We will call this more descriptive phase of the inference **Step 1** of the standard reasoning in the following. (b) inferred that the accuracy in the indirect task was better than in the direct task. We will use the purely descriptive term **indirect task advantage ([ITA]{})** to refer to a situation with a higher accuracy (or, more generally, sensitivity) in the indirect as compared to the direct task.

Based on their claim of an [ITA]{}, proceed to interpret this [ITA]{} with relation to conscious vs. unconscious processing (we will call this more interpretative phase of the inference **Step 2** of the standard reasoning in the following): They assume, that the direct task measured mainly conscious processes (because it was essentially a report of what participants believe), while the indirect task measured mainly unconscious processes (because the masked pictures were hardly consciously seen by the participants, as confirmed by an additional control experiment in which participants had to determine whether the masked suspects were male or female). Based on these assumptions, relate the direct task to conscious and the indirect task to unconscious processing, thereby attributing the supposed [ITA]{} from Step 1 to superior unconscious processing: “although humans cannot consciously discriminate liars from truth tellers, they do have a sense, on some less-conscious level, of when someone is lying” (p. 1103) and “the unconscious mind identifies and processes cues to deception (to the extent that they are available) more efficiently and effectively than the conscious mind.” (p. 1104).

However, —and the standard reasoning in general—inferred an [ITA]{} (i.e., better accuracy in the indirect than in the direct task) although they never calculated an accuracy for the indirect task. They only showed that there was a congruency effect on RTs in this task. We will show that even a very clear and highly significant congruency effect is not indicative of good accuracy (or, more generally, sensitivity). Because laudably followed an open-data policy, were able to reanalyze these data and to assess how much evidence the congruency effect truly provided for relatively good accuracy in the indirect task. Their argument went as follows: **If the congruency effect is supposed to indicate relatively good accuracy for deception detection, then we must be able to use the corresponding data to determine who of the suspects is lying.** Otherwise the data cannot provide evidence for relatively good accuracy in the indirect task.

To assess this, determined statistically optimal classifiers, used the RTs of the indirect task to classify (“predict” in the nomenclature of statistical learning) who of the suspects were lying, and found the accuracy of the indirect task to be only at 50.6%-correct ($SEM = 0.3\%$; see below for more details on the methods used). This value is very similar to and not significantly different from the accuracy in the direct task (49.62%-correct; $SEM = 1.4\%$). Therefore, ’s inference in Step 1 was flawed: The data did not provide evidence for better accuracy in the indirect than in the direct task: There was no [ITA]{}. Because the existence of an [ITA]{} in Step 1 is a necessary condition for Step 2, these inferences about unconscious vs. condition processing were also not warranted.

General description of standard reasoning {#standardReasoning .unnumbered}
-----------------------------------------

![**Typical study design to infer an indirect task advantage ([ITA]{}).** **(a)** In each trial, prime and target–stimuli are presented. The prime–stimulus is masked such that it appears hardly visible to the participants. Prime and target can be either from category A or B. In the direct task, participants discriminate the *prime* (e.g., decide whether it is from category A or B) and perform poorly. In the indirect task, the same stimuli are presented and participants judge whether the *target* is from category A or B. Now, the prime does clearly influence responses. For example, participants respond faster if prime and target are from the same category (congruent: A–A, or B–B) than if they are from different categories (incongruent: A–B, or B–A). Based on this congruency effect, the standard reasoning infers better sensitivity in the indirect than in the direct task (a situation we dubbed [ITA]{}) and attributes this better sensitivity to unconscious processing. **(b)** A typical example is : Prime and target stimuli were numbers that were either smaller or larger than five. In the direct task, participants discriminated the primes and their sensitivity was poor. In the indirect task, participants classified the targets and showed faster responses and larger lateralization of brain activity if prime and target were congruent than if they were incongruent. followed the standard reasoning, inferred an [ITA]{}, and concluded that the primes were unconsciously processed in the absence of conscious awareness.[]{data-label="fig:ITAdesign"}](figures/prototypical_ITA_paradigm.pdf){width=".7\textwidth"}

Let us now describe the standard reasoning (and its fallacy) in a more general way. The typical unconscious priming paradigm encompasses two tasks. In the **direct task**, participants are presented with masked stimuli belonging to one of two categories (labeled A and B in Figure \[fig:ITAdesign\]a). In each trial, participants perform a binary discrimination. Performance is measured by the sensitivity index $d'$ from Signal Detection Theory [@Green_Swets_88] or by %-correct. The stimuli are masked such that performance in the direct task is typically poor ($d'$ close to zero or %-correct close to $50\%$). Because participants directly classify the masked stimulus, this task is called direct task.

![**Standard reasoning to infer an indirect task advantage ([ITA]{}).** **(a)** In the *direct task*, the standard reasoning infers from close-to-chance performance that there was poor sensitivity for the primes, if any at all. **(b)** In the *indirect task*, the standard reasoning infers from a clear congruency effect that sensitivity for the primes was relatively good. Based on this pattern of results the standard reasoning makes two inference steps: In Step 1, it incorrectly infers that participants’ responses in the indirect task were more sensitive to the primes than in the direct task ([ITA]{}). In Step 2, it attributes this difference to unconscious processing. However, already Step 1 of this reasoning is flawed because a clear congruency effect does not indicate good sensitivity. It could be caused by a sensitivity as poor as (or even worse than) the sensitivity in the direct task! Because Step 1 of the standard reasoning is independent of any (sometimes contentious) assumptions about conscious vs. unconscious processing, our critique is also independent of any such assumptions.[]{data-label="fig:ITAStandardReasoning"}](figures/ITATwoMeasures.pdf){width="70.00000%"}

In the **indirect task**, the same masked stimuli are shown (in this task often called “primes”), but participants typically respond to subsequent “target” stimuli (Figure \[fig:ITAdesign\]a). As a result, the primes clearly affect responses to the targets. For example, if the prime is congruent to the target (both from category A or both from B) participants’ RTs are faster than if they are incongruent (one from category A, one from B). Or, similarly, brain activity could be different between congruent and incongruent trials. These **congruency effects**, are the basis for the following inferences of the standard reasoning. Because in this task, the influences of the primes are only indirectly measured (the primes are not task relevant), this task is typically called the indirect task.

From such a pattern of results, the standard reasoning infers unconscious processing in two steps (Figure \[fig:ITAStandardReasoning\]): In **Step 1**, it infers that (a) sensitivity for the primes in the direct task was very poor, if not zero, (b) sensitivity for the primes in the indirect task was relatively good and, therefore, better in the indirect task than in the direct task, a situation we dubbed [ITA]{}.

In **Step 2**, the standard reasoning associates the direct task with conscious processing and—based on the supposed [ITA]{} from Step 1—concludes that the primes have been processed to an extend beyond what participants can consciously report, such that the primes were processed outside of conscious awareness.

For example, summarize the standard reasoning like this: “The present work follows the tradition of providing evidence for a dissociation between direct and indirect effects of unconsciously presented stimuli (Greenwald, Klinger & Schuh, 1995; Draine & Greenwald, 1998). More specifically, null effects are sought in direct measures (i.e. where subjects respond directly to the unconsciously presented stimuli) accompanied by non-null indirect effects (i.e. priming effects)” (p. B2). Note that an [ITA]{} in Step 1 of the standard reasoning is sometimes only implicitly inferred but it is always necessarily implied when claims on unconscious processing are made. In the following sections, we describe why the claims of an [ITA]{} in Step 1 of the standard reasoning is flawed. Note, that our critique is independent of assumptions about conscious vs. unconscious processing that have to be made in Step 2 and for which different and sometimes contentious approaches exist (e.g., ). This makes our critique very general.

Standard reasoning is flawed and cannot provide evidence for an [ITA]{} {#sec:examples .unnumbered}
=======================================================================

The standard reasoning is intuitively very appealing, which seems to be one reason for its popularity. The colloquial version of the arguments in Step 1 of the standard reasoning that lead to claims of [ITA]{}s goes like this: “Participants have a very hard time to discriminate the primes in the direct task. They are very close to zero sensitivity and usually not significantly above chance. Nevertheless we can easily find clear and highly significant congruency effects in the indirect task. Therefore it seems obvious, that the indirect task responses are more sensitive to the primes than those in the direct task.”

However, this intuition is misguided. A clear congruency effect could as well be compatible with an underlying true accuracy in the indirect task of $51\%$-correct as with $99\%$-correct. We first demonstrate the problem by using a toy example, then discuss it in a general setting.

Toy example with birth weights of babies {#sec:toyExampleBabies .unnumbered}
----------------------------------------

In this example, we focus exclusively on the analysis strategies in the direct and indirect tasks. We show that even if responses in the direct and indirect tasks were based on the exact same information, the standard reasoning would erroneously infer that responses in the indirect task were *more* sensitive than responses in the direct task.

Consider we told participants the birth weights of newborn girls (category A) and boys (category B) and that participants had no other information about the babies. In the “direct task”, participants used this weight information to guess whether a baby is a girl or a boy (girls weigh on average a little less than boys). Due to the large overlap between the weight distributions (Figure \[fig:toyExample\]a), participants would be correct only approx. $55\%$ of the time, which is a poor performance and close-to-chance level (50%). Following the standard reasoning, an experimenter would correctly infer a poor sensitivity when it comes to telling apart baby girls and boys using their weight.

![ **Toy-example demonstrating fallacy of standard reasoning.** We show that responses in the direct and indirect tasks can be based on the exact same information while nevertheless the standard reasoning would infer an indirect task advantage ([ITA]{}). For better intuition, we chose real-world data [@janssen2007standards]: Newborn girls weigh a little less than boys. Consider participants of a hypothetical experiment knew the weight, but nothing else about the babies. **(a)** In the *direct task*, participants used the weight of an individual baby to guess whether it is a girl or a boy. The weight distributions overlap heavily leading to poor sensitivity regarding this task ($d'_\text{true} = 0.25$; corresponding to $55\%$-correct). **(b)** In the *indirect task*, consider participants’ reaction times (RTs) reflected a baby’s weight: A weight of 3500 g corresponded to an RT of 350 ms, 3600 g to 360 ms, 3700 g to 370 ms, etc. Participants would be presented with many babies, and the experimenter would calculate the mean RTs. In a typical experiment, this would be the average across 10 participants and 300 trials in each condition, which is approximately equivalent to 3000 girls and 3000 boys. These group means are much less variable than the individual values, such that the experimenter would obtain a clear separation between the two means (this corresponds to a clear congruency effect in the priming paradigm). Based on this result, the standard reasoning would erroneously infer that participants had relatively good sensitivity about whether a baby was a girl or a boy in the indirect task—better than in the direct task. That is, the standard reasoning would infer an [ITA]{} even though the *exact same* information created the responses in both tasks. []{data-label="fig:toyExample"}](figures/figureToyExample.pdf){width="80.00000%"}

In the “indirect task”, participants performed some response (it does not matter here which response exactly), and the weight information would somehow affect participants’ RTs in such a way that a weight of 3500 g led to an RT of 350 ms, 3600 g to 360 ms, etc. Many baby weights would be presented to the participants and—if enough samples were presented—the experimenter would eventually find a clear separation between the mean RTs of the participants to groups of baby girls and boys. Figure \[fig:toyExample\]b shows this for $3000$ observations in each group, which is a typical number of observations in the indirect task (= number of participants $\times$ number of trials per participant).

Here is the catch: Following Step 1 of the standard reasoning, the experimenter would interpret this clear separation of mean RTs as evidence that there was a relatively good sensitivity in the indirect task about whether a baby is a girl or a boy (Figure \[fig:ITAStandardReasoning\]b)—better than in the direct task. But this inference is wrong, because exactly the same information was the basis for the responses in both tasks.

Where is the flaw? The problem is that the standard reasoning calculates two very different things in the direct and indirect tasks: In the direct task, it calculates how good each individual baby can be classified based on the weight information. In the indirect task, it assesses whether there is a difference in mean responses to groups of baby girls and boys. These are two very different things and it is a-priori to be expected that the discrimination at the level of individual babies can be poor while the mean responses to a group of babies can nevertheless be clearly separated between the categories.

Our toy example demonstrates that Step 1 of the standard reasoning is flawed because it infers [ITAs]{} even when there are none. In the next section we will describe the fallacy more generally.

General description of the fallacy {#general-description-of-the-fallacy .unnumbered}
----------------------------------

The main fallacy of the standard reasoning is that it infers an [ITA]{} even when the true sensitivity is exactly the same in the direct and indirect tasks. Consider that the same, poor information underlied both tasks, represented internally by two normal distributions with slightly different means. In the direct task, the measured sensitivity would be close to zero, as is typically found. Nevertheless, the same information can cause slightly different mean responses in the congruent and incongruent conditions of the indirect task and those means will be clearly separated when we use enough trials. This is so because the variability of the means becomes smaller with more observations. Therefore, a clear congruency effect is not indicative of good sensitivity: It could have been caused by a good sensitivity (corresponding to, say, $d' = 5$ or 99%-correct) as well as by a very poor sensitivity (say, $d' = 0.05$ or 51%-correct). Not recognizing this is the **main fallacy of the standard reasoning**.

To see this, consider what happens if we increased the number of trials (be it by increasing the trials per participant or by increasing the number of participants). The direct task sensitivity will only be measured more precisely but will keep estimating the true, poor sensitivity. In contrast, the indirect task’s congruency effect becomes clearer as the mean distributions become more separated (due to the variability of the means becoming smaller).

To overcome this fallacy, we would need to estimate sensitivity in both tasks in a similar way. Our analyses will do this by estimating in both tasks the separation of the underlying distributions relative to the standard deviation at a single trial level (this corresponds to the well–known sensitivity index $d'$ from Signal Detection Theory; ). However, before being able to describe the appropriate analysis, we need to discuss some details related to the fallacy of the standard reasoning.

### True zero–sensitivity in direct task {#true-zerosensitivity-in-direct-task .unnumbered}

Consider the true sensitivity in the direct task were known to be indeed exactly zero and there were at the same time a clear congruency effect in the indirect task. In this case (and only in this case), we could indeed justify the claim that the indirect task were somehow more sensitive than the direct task. This is so, because any—even minutely—positive sensitivity required to produce the congruency effect is larger than zero. However, there are a number of problems with this scenario: (a) It is unrealistic. Typically studies either find some small, residual sensitivity in the direct task or they do not find a clear congruency effect. (b) The sensitivity in the indirect task could still be so low, that it would be close–enough to the zero sensitivity of the direct task to not allow for strong conclusions (e.g., consider a sensitivity that corresponded to 50%-correct in the direct task and to 51%-correct in the indirect task). (c) If we take into account that the sensitivity in the direct task always needs to be measured (and therefore is affected by measurement error), we would still need to establish that the sensitivity in the indirect task is indeed larger than that in the direct task (e.g., by a significance test on the difference, as is performed by our more appropriate methods but not by the standard reasoning).

### The role of significance testing {#the-role-of-significance-testing .unnumbered}

Until now, we purposefully did not talk much about statistical significance testing because we wanted to focus on the main fallacy of the standard reasoning. This fallacy is independent of details of significance testing. Significance testing can be seen as only a tool to establish that there is a “clear congruency effect” or that a sensitivity is “close to zero”. Because significance testing and its applications have been heavily—and often rightfully—criticized since the very inception of the concept [@Boring_1919; @Morrison_Henkel_70; @Dienes_11; @Cumming_14], it might be tempting to attribute the main fallacy of the standard reasoning also to significance testing. However, the main fallacy of the standard reasoning would persist even if we replaced the significance tests by other methods; for example, by corresponding Bayesian methods. This is so because the standard reasoning asks two very different statistical questions in the direct and indirect tasks. Simply answering those questions with other statistical methods would therefore not ameliorate the main fallacy of the standard reasoning. Throughout this article, we therefore followed a pragmatic approach by using statistical significance testing while avoiding typical and well–known errors when applying those methods [@Cumming_14].

### Correct uses of significance testing {#correct-uses-of-significance-testing .unnumbered}

Many of the studies we analyzed show problematic uses of significance testing. The main problem is created by an often underpowered direct task: A non-significant result does not guarantee that the true sensitivity is zero. There can easily be a poor but above zero sensitivity, which was undetected due to low statistical power, which is often prevalent in such studies [@vadillo2020unconscious]. The congruency effect in the indirect task can then be the result of this very same, poor sensitivity and nevertheless reach significance because of typically much higher statistical power in the indirect task. Therefore, if we are interested in establishing a difference between sensitivities in both tasks, it does not suffice to evaluate both tasks in isolation. We must test directly whether there is a difference between the two tasks. Failure to do so can lead to serious errors (cf. ; and appendix B of for another reanalysis showing how serious these problems can be). We show in the next section how testing for the difference is an integral part of the appropriate analysis.

Appropriate analysis: Calculate sensitivities and test for difference {#sec:correctAnalysis .unnumbered}
=====================================================================

We first describe the appropriate analysis, assuming that trial–by–trial data are available. This is consistent with the analysis used by . Then we describe our newly developed method to analyze studies when only summary statistics are available.

Appropriate analysis when trial–by–trial data are available {#appropriate-analysis-when-trialbytrial-data-are-available .unnumbered}
-----------------------------------------------------------

The standard reasoning infers differences in sensitivities even when there are none. The appropriate method must properly compare sensitivities in the direct and indirect tasks. The analysis steps for both tasks must be equated such that the calculated statistics are comparable, followed by a test of the difference between the two tasks. Similar approaches have been used in previous—albeit very few—studies [@Dulaney_Eriksen_59; @Klotz_Neumann_99; @kunst1980affective; @Schmidt_02; @Franz_vonLuxburg_15_PS] in accordance with the long standing (but often ignored) request to use the “same metric” for both tasks [@Reingold_Merikle_88].

In both tasks, we compute $d'$ using Signal Detection Theory [@Green_Swets_88] and test for a difference between the two tasks. In the [**direct task**]{}, participants typically classify the primes in each trial and a $d'$ value is computed as it is also often done in studies using the standard reasoning. In the [**indirect task**]{}, however, the standard reasoning relies on the congruency effect in continuous measures (e.g., RTs or brain activity as measured by EEG or fMRI). For a proper comparison, we have to transform these continuous measures into a classification for each trial. This can be achieved in different ways, and we suggest to use the optimal classifier for the given setup. This gives the indirect task the best possible performance thereby increasing the chances of finding an [ITA]{} and making our arguments stronger when we nevertheless do not find an [ITA]{} (Box \[box:conservativeAnalysis\]).

Which classifier is best? We have shown that under typical conditions with equal number of congruent and incongruent trials, the median-split classifier is optimal (; see Appendix \[app:medianClassifier\] for details and proof). The classification proceeds as follows: For each participant, we determine the median RT and classify (“predict” in the nomenclature of statistical learning) all trials with smaller RTs as congruent, and trials with larger RTs as incongruent. Then we compare these classifications to the true labels (congruent/incongruent) evaluating for each trial whether the classification was correct or not, and we then compute a $d'$ value as in the direct task. Finally, we compare the $d'$ values of the direct and indirect task and test for a difference.

Some details: (a) Instead of computing $d'$ values, the analysis could also be based on %-correct values. Assuming an unbiased observer predicting both categories equally often in the direct task, both approaches produce the same results and we later report both measures to foster intuition. (b) Dichotomization of the continuous, indirect measures will result in a loss of information [@Cohen_83]. However, the direct task also requires participants to give binary responses. That is, dichotomization is also forced in the direct task by the experimental design. Since participants plausibly have a continuous sense (confidence) about a stimulus’s category [@rausch2018confidence; @zehetleitner2013being], the binary response format in the direct task also results in an information loss. Our approach only equates this dichotomization. Because we choose the best possible way to dichotomize, we increase chances of finding an [ITA]{} (Box \[box:conservativeAnalysis\]). (c) We classify the trials of the indirect task according to the labels congruent/incongruent and not according to the prime category A/B, as is typically asked in the direct task. This is so because studies typically find a congruency effect between prime and target (and not a mere effect of the prime being in category A or B). For a comparison to the direct task, we would ideally transform the congruency classification into a classification of the prime (A vs. B). For simplicity, we assume an optimal transformation here (without errors). This is plausible, because the target stimuli are typically fully visible to the participants, such that errors are rare. Again our approach gives the indirect task the best possible performance, increases the chances to find an [ITA]{}, and makes our argument stronger when we nevertheless do not find an [ITA]{} (Box \[box:conservativeAnalysis\]).

Appropriate analysis when only summary statistics are available {#sec:method .unnumbered}
---------------------------------------------------------------

Because the standard reasoning to infer an [ITA]{} is flawed, many already published studies on unconscious priming need reassessment. However, the appropriate analysis as described in the previous section would require full trial-by-trial data. These can be difficult or impossible to obtain for published studies [@wicherts2006poor]. For the older—but nevertheless influential—studies, those data might not even exist anymore. Therefore, we developed an approach that allows to estimate the results of the appropriate analysis without access to trial-by-trial data and solely based on the typically reported statistics. Here, we sketch the central approach of this analysis; details are given in Appendix \[app:reanalysisFormulas\].

The overall aim of this reanalysis is, again, to estimate sensitivities for the direct and indirect tasks (i.e., to either calculate $d'$ from Signal Detection Theory or %-correct assuming a neutral observer). The direct task typically already provides $d'$ or %-correct values. In the indirect task, studies typically report $t$ or $F$ values from a repeated measures design for the congruency effect. For these repeated measure designs, we derive an estimator for the underlying sensitivity. Since this reanalysis can only use the reported statistics, there is one free parameter that needs to be estimated: the ratio of between– vs. within–subject variances, which we denote by $q^2$. We estimated this parameter based on (a) our own replication experiment (b) an extensive literature review, and (c) extensive simulations (see Appendix \[app:roleOfq\]). Assuming the largest plausible value for $q^2$, we again maximize the estimated sensitivity in the indirect task and therefore increase the likelihood of finding an [ITA]{} (Box \[box:conservativeAnalysis\]).

Replication of landmark study finds no [ITA]{} {#replication-of-landmark-study-finds-no-ita .unnumbered}
==============================================

We are now equipped with the appropriate tools that allow to analyze typical settings and tasks that have been investigated in the context of unconscious priming. In this section, we will focus on one highly influential study on unconscious semantic priming of numbers [@Dehaene_etal_98]. We will first describe the study and how its conclusions depend crucially on the standard reasoning. Then, we will describe a replication experiment of the behavioral part of this study and analyze the trial–by–trial data. In the next section we will then reanalyze the published data of this study (besides other studies). Overall, we will conclude that the results of our replication–experiment are similar to those of the original study and that both give reason to seriously doubt that there is an [ITA]{} for semantic priming of numbers as inferred by .

were interested in the question of whether the semantic meaning of numbers can be processed outside conscious awareness. They employed a prototypical version of the standard reasoning (Figure \[fig:ITAdesign\]b): In the direct task, participants discriminated features of masked numbers and performed poorly ($d'=0.2$; corresponding to $54\%$-correct). In the indirect task, participants were again presented with the masked numbers (now serving as primes), but decided whether subsequent target numbers were smaller or larger than five. Participants responded app. $24$ ms faster when prime and target were congruent (both larger or smaller than five) than when they were incongruent (one smaller and one larger than five). Similar congruency effects were found for brain activity in EEG and fMRI (i.e., larger lateralization of brain activity in congruent than incongruent trials).

interpreted these results according to the standard reasoning: In **Step 1**, they inferred an [ITA]{}: larger sensitivity in the indirect task than in the direct task: “\[participants\] could neither reliably report \[the prime’s\] presence or absence nor discriminate it from a nonsense string \[...\] Nevertheless, we show here that the prime is processed to a high cognitive level.” (p. 597). In **Step 2**, they argued that “the prime was unconsciously processed” (p. 597) because participants were at chance performance in the direct task. Overall, they concluded: “By showing that a large amount of cerebral processing, including perception, semantic categorization and task execution, can be performed in the absence of consciousness, our results narrow down the search for its cerebral substrates” (p. 599). In short, employed a prototypical version of the standard reasoning to infer an [ITA]{} and unconscious processing; exactly as described above. To assess the validity of these claims, we first attempted to replicate the behavioral part of that study, later we will reanalyze the published data of the study.

Disclosures {#disclosures .unnumbered}
-----------

### Data, materials, and online resources {#data-materials-and-online-resources .unnumbered}

The experimental material, data and the scripts for the analyses reported in this article will be made available on the Open Science Framework (OSF), at https://osf.io/kp59h.

### Reporting {#reporting .unnumbered}

We report how we determined our sample size, all data exclusions, and all measures in the study.

Methods {#methods .unnumbered}
-------

Twenty-four volunteers participated (13 female, 5 left-handed, age range: 19–27 years; $M = 21.5$, $SD = 1.9$). All had normal or corrected-to-normal vision, signed written informed consent and were naïve to the purpose of the experiment. In the original study, six and seven participants took part in the first and second direct task, respectively, and 12 participants took part in the indirect task. We took great care to make stimuli and timings as similar as possible to those of the original study by . Each trial consisted of: fixation cross ($417$ ms), forward mask ($67$ ms), prime ($42$ ms), backward mask ($67$ ms), and target ($200$ ms). In the original study, those values were: forward mask ($71$ ms), prime ($43$ ms), backward mask ($71$ ms), and target ($200$ ms; cf. Figure \[fig:ITAdesign\]b). Slight differences in timing are due to slightly different refresh rates of the monitors used. The prime duration of $43$ ms was chosen by the original authors because it was the longest duration that produced non-significant results in the direct tasks. Primes and targets were numbers (1, 4, 6 or 9) that were either presented as digit (e.g., “1”) or word (e.g., “EINS”; German for “ONE”). The original study used the same numbers but not in German. As in the original study, primes and targets could be congruent (both smaller or both larger than 5) or incongruent (one smaller, one larger). Masks were composed of seven randomly drawn characters from $\{\text{a-z, A-Z}\}$. Participants were seated in front of a monitor (VIEWPixx /3D, VPixx Technologies Inc., Montreal, Canada), effective refresh rate $120$ Hz at a viewing distance of approx. $60$ cm in a sound- and light-protected cabin. In the original study the monitor was a cathode-ray tube (CRT) with a refresh rate of $70$ Hz. Stimuli were presented centrally as white text ($69$ cd/m$^2$; character height: $1^\circ$; width: $0.5^\circ$ visual angle; font: Helvetica) on a black background ($0.1$ cd/m$^2$). In the original study those luminance values were not specified so that we chose the most plausible settings for our experiment.

In the direct task, participants classified whether the prime was smaller or larger than five. We used this particular task because argued that it is “better matched with the \[indirect\] task” (p. 227). In the original study by , two direct tasks were used, that produced similar results: In their first direct task, the prime stimulus was omitted in some trials and participants had to discriminate their presence vs. absence. In the second direct task, the prime stimuli were replaced by random letter strings and participants had to discriminate between numbers vs. random strings. In the indirect task, participants decided as quickly as possible whether the target was smaller or larger than five; as was the case in the original study. Each participant performed $256$ trials per task, preceded by $16$ practice trials in each task. In the original study, participants performed only $96$ and $112$ trials in the first and second direct task, respectively, and $512$ trials in the indirect task. We repeated indirect task trials with RTs that were too fast ($<100$ ms) or too slow ($>1$ s). Similarly, the original study rejected too fast ($<250$ ms) and too slow trials ($>1$ s). The indirect task was performed before the direct task (as is common practice in this paradigm) to prevent participants attending to the prime in the indirect task. In the original study the direct and indirect tasks were performed by different groups.

Number of participants and trials were chosen to produce a statistical power of above $95$% to find a difference between sensitivities and confirm an [ITA]{} if it is there (see Appendix \[app:validation\]). For this power estimation we assumed a true sensitivity of $d'_\text{true} = 0$ in the direct task vs. $d'_\text{true} = 0.25$ in the indirect task (values based on our reanalysis of , see below). To our knowledge, the original study did not perform a power analysis. A post hoc power analysis revealed that the original study had a statistical power of only $46$% to find an [ITA]{} given that direct and indirect sensitivities are $d'_\text{true} = 0$ and $d'_\text{true} = 0.25$, respectively. This low power is due to a small number of direct task participants and trials.

Results and Discussion {#sec:replicationDehaene .unnumbered}
----------------------

Our analysis proceeded in two strands: First the traditional analysis which forms the basis for the standard reasoning, then the appropriate analysis.

### Traditional analysis {#traditional-analysis .unnumbered}

The direct task sensitivity was $d' = 0.26$ ($SD = 0.27$), $t(23) = 4.81$, $p < .001$, corresponding to an accuracy in prime identification of $M=54.87$%-correct ($SD = 4.9$, $t(23) = 4.82$, $p < .001$). This is exactly in the range of sensitivities found in their direct tasks ($d' = 0.3$ in the first and $d'=0.2$ in the second direct task). For a graphical depiction of these results, compare the bars corresponding to the direct tasks in Figures \[fig:replicationComparison\]a and \[fig:replicationComparison\]b. Note, however, that our direct task sensitivity is significantly above zero, while this was not the case in the original study ($p>.05$). This is so, because we sampled much more participants and trials than in the original study. Therefore, we had much higher statistical power. To simulate the lower power of the original study, we discarded data from participants and trials to match the same number of observations as in the original study: We kept only the first $N=7$ participants and the first 112 trials of each participant. This leads to a non-significant result, $d' = 0.31$ ($SD=0.39$), $t(6) = 2.06$, $p = 0.086$, as was the case in the original study. Therefore, it is plausible that it was the low statistical power in the original study (and not the sensitivity being exactly at zero) that was the reason for the non–significant result in the direct task of the original study. This interpretation is also corroborated by the fact that we measured numerically essentially the same sensitivity in this task as the original study.

In the indirect task, the congruent condition yielded faster RTs ($M = 445$ ms, $SD = 42$) than the incongruent condition ($M = 457$ ms, $SD = 37$), resulting in a clear and highly significant congruency effect of $M=12$ ms ($SD = 11.8$), $t(23) = 4.95$, $p < .001$. The correlation of mean RTs in congruent and incongruent conditions across participants was $r=.96$, $95$% CI $[.91,.98]$. That is, we found a highly significant congruency effect on RTs, as did the original study.

There is one potential caveat here, though. Our congruency effect was lower than that in the original study ($12$ ms vs. $24$ ms, respectively). We will discuss this when we describe the reanalysis of the published data of .

In summary, we found a similar pattern of results as in the original study: A very poor direct task performance and a clear congruency effect in the indirect task. Based on this pattern of results many researchers would have applied the standard reasoning and inferred an [ITA]{}.

![**Sensitivities in the paradigm**. **(a)** Results of our replication study, based on our full trial-by-trial reaction time (RT) data. **(b)** Results of our reanalysis approach based on the published statistics for the RT data from . **(c)** Reanalysis results from digitizing Figure 2b from showing histograms of indirect task’s RT data. For the comparison, we used the same direct task results as in subfigure (b). Comparing (a)–(c) we see that our replication closely matches the results of the original study. **(d)** Difference in sensitivities between direct and indirect tasks: Neither in our replication study nor in our reanalyses of there is a significant difference in sensitivities. That is, there is no indication of an [ITA]{}. The reanalysis result from (b) is also shown in the large summary in Figure \[fig:reanalysis\]. Error bars indicate 95% confidence intervals.[]{data-label="fig:replicationComparison"}]({figures/replicationComparison}.pdf){width="100.00000%"}

### Appropriate analysis {#appropriate-analysis .unnumbered}

The appropriate analysis compares sensitivities in direct and indirect tasks. We have already described in the last section that the direct task in our experiment yielded a sensitivity of $d' = 0.26$ ($SD = 0.27$), corresponding to an accuracy of $M=54.87$%-correct. For the indirect task, we obtained a sensitivity of $d'=0.25$ ($SD = 0.15$), corresponding to an accuracy of $M=54.93$%-correct ($SD = 3.03$). These values are depicted in Figure \[fig:replicationComparison\]a. Inspecting the figure shows that sensitivities for direct and indirect tasks are very similar, as also depicted by the left bar of Figure \[fig:replicationComparison\]d. We found virtually no difference between these sensitivities, $M = -0.01$ ($SD=0.34$), $t(23) = -0.2$, $p = 0.844$. The correlation between sensitivities in both tasks across participants was $r=-.17$, $95$% CI $[-.54,
.24]$. That is, there is no indication for an [ITA]{}

In conclusion, using the traditional analyses, our results are similar to the typical pattern of results found by and many researchers would have inferred an [ITA]{}. However, the appropriate method yields no evidence for an [ITA]{}: The sensitivities in both tasks are essentially identical.

Reanalysis of 15 influential studies finds hardly any [ITA]{} {#reanalysis-of-15-influential-studies-finds-hardly-any-ita .unnumbered}
=============================================================

After having demonstrated that the problems of the widely used standard reasoning are indeed serious, we now applied our approach to a sample of 15 highly relevant studies in the field of unconscious priming.

Selection criteria for reanalyzed studies {#selectionCriteriaForStudies .unnumbered}
-----------------------------------------

We focused on studies that used the standard reasoning and claimed an [ITA]{}. First, we selected eight studies by hand that are particularly relevant and are based on the standard reasoning. These studies and their number of citations in Web of Science (Clarivate Analytics, Philadelphia, U.S.A.) are: , , , , , , , ,

Second, we searched for English articles in Web of Science with the topic “unconscious priming”. We selected all studies with more than 150 citations. This resulted in seven additional studies: , , , , , , . Overall, these 15 studies received a total of [3277]{} citations. See Appendix \[app:reportedResults\] for details on these studies.

Details of analysis when only summary statistics are available {#sec:method .unnumbered}
--------------------------------------------------------------

Our reanalysis method estimates sensitivities for direct and indirect tasks. Here, we sketch the technical details of the analysis. A detailed account with mathematical derivations is given in Appendix \[app:reanalysisFormulas\].

We denote the estimated sensitivities in the direct and indirect tasks by $d'_\text{estimated,direct}$ and $d'_\text{estimated,indirect}$, respectively. For the direct task, the typically reported statistics are average $d'$ or %-correct values. Therefore, our estimate is simply the measured sensitivity, $$d'_\text{estimated,direct} = d' ,$$ or a well–known conversion of %-correct values to $d'$ values assuming unbiased observers [@Green_Swets_88], $$d'_\text{estimated,direct} = 2\Phi^{-1}(\text{\%-correct}),$$ where $\Phi^{-1}$ is the inverse of the normal cumulative density function.

In the indirect task, statistics for the congruency effect are typically reported by $t$ values from a paired $t$ test or $F$ values from a repeated–measures ANOVA. In this setting, $F$ values can be translated into $t$ values by $|t| = \sqrt{F}$. From a $t$ value, we estimate the sensitivity by $$d'_\text{estimated,indirect} = t\cdot \kappa_{N,M,q^2} ~~~~\text{with}~~~~~ \kappa_{N,M,q^2}
        =
        \frac{\Gamma\left(\frac{N-1}{2}\right)}{
        \sqrt{\frac{N-1}{2}}\Gamma\left( \frac{N-2}{2} \right)}\sqrt{\frac{Mq^2 + 2}{NM}}.$$ where $\Gamma$ is the gamma distribution. The constant $\kappa_{N,M,q^2}$ corrects for the fact that $t$ values increase with increasing number of participants ($N$), increasing number of trials ($M$), and that they depend on the ratio of between- and within-subject variance, which we denote by $q^2$. The parameter $q^2$ is the only parameter we need to estimate for our approach. It is reasonable to assume that this ratio is at most $q^2 = 0.0225$ given our replication study and given an extensive literature review and simulations (see Appendix \[app:roleOfq\]). Assuming the largest plausible value for $q^2$, we again maximize the estimated sensitivity in the indirect task and therefore increase the likelihood of finding an [ITA]{} (Box \[box:conservativeAnalysis\]).

From the estimated sensitivities, we compute the difference $$d'_\text{difference} = d'_\text{estimated,indirect} - d'_\text{estimated,direct}$$ and construct a 95% confidence interval using the corresponding standard errors. This allows to test for an [ITA]{}: If the confidence interval lies above $0$ (that is, it has the form $[a,b]$ with $a>0$), the reported result is significant and an [ITA]{} is confirmed, otherwise there is not sufficient evidence to claim an [ITA]{}. See Appendix \[app:validation\] for an extensive validation of this reanalysis method.

![**Reanalysis of influential studies reporting indirect task advantages ([ITAs]{}).** The 15 studies used the standard reasoning and inferred 44 [ITAs]{}. **(a)** Sensitivities according to our reanalysis. To foster intuition, we also show %-correct values assuming a neutral observer. **(b)** Differences between indirect minus direct sensitivities: In each group of bars, the indirect task is compared to the corresponding direct task. Only if a confidence interval lies to the right and does not contain $0$, there is evidence for an [ITA]{}. Inspection shows that only in very few cases (8 out of 44), there is a significant [ITA]{}, while in most cases there is no evidence for an [ITA]{} (35 out of 44), and there is even one case with a significant opposite result, an advantage of the direct task (DTA). Not a single study provides consistent evidence for an [ITA]{} across its experiments and conditions although the studies claimed [ITAs]{} for all reanalyzed experiments and conditions. This pattern of results casts serious doubts on the existence of [ITAs]{} in most, if not all, of the studies. Error bars represent 95%-confidence intervals. []{data-label="fig:reanalysis"}](figures/reanalysisVisualization.pdf){width="\textwidth" height="19.25cm"}

Results and Discussion {#results-and-discussion .unnumbered}
----------------------

We first describe our reanalysis in detail for the study of and then use the same methods for all the other studies.

### Reanalysis of  {#reanalysis-of .unnumbered}

The study reported two direct tasks with sensitivities of $d' = 0.2$ and $d' = 0.3$, respectively. We used the results of the first task, because it had the smaller sensitivity, thereby increasing again the chances of confirming an [ITA]{} (Box \[box:conservativeAnalysis\]). In this direct task, $N=7$ participants were sampled in $M = 56$ trials per condition (a total of 112 trials per participant). Taken together, our reanalysis method estimates from these values the direct task sensitivity and its standard error to be $d'_\text{estimated,direct} = 0.20$ ($SE = 0.11$), as depicted in Figure \[fig:replicationComparison\]b. In the indirect task, the study reported on average a congruency effect of $24$ ms with a standard deviation of $13.5$ ms in a sample of $N=12$ participants sampled in $M=256$ trials per condition (totaling 512 trials). This equals a $t$ value of $t = 24~ms/(13.5~ms/\sqrt{12}) = 6.12$ from which our reanalysis method estimates the sensitivity to be $d'_\text{estimated,indirect} = t \cdot \kappa_{N,M,q^2} = 0.29$ ($SE = 0.09$), where we assume $q^2 = 0.0225$ and, thus, $\kappa_{12,256,0.0225} = 0.047$, as again depicted in Figure \[fig:replicationComparison\]b.

Inspecting Figure \[fig:replicationComparison\]b shows that the sensitivities in both tasks are very similar, $d'_\text{difference} = 0.09$, $SE = 0.14$ (cf. Figure \[fig:replicationComparison\]d). The confidence interval for the difference includes zero, $95\%~$CI$~=[-0.18, 0.35]$, thereby indicating that the difference is not significantly different from zero. That is, there is no evidence for an [ITA]{}.

We were able to reanalyzed the results from in an additional way. They depicted summary histograms of RTs in their Figure 2b visualizing that congruent and incongruent RT distributions are similar in shape; with incongruent RTs being shifted to the slower RTs as compared to congruent RTs. Nevertheless, RT distributions largely overlap. We digitized the histogram and split RTs along the median as described in the appropriate analysis section. From this, we estimated the indirect task sensitivity to be $d'=0.23$ ($SE
= 0.03$). Again, there is no difference to their first direct task’s sensitivity ($d' = 0.2$, $SE = 0.11$) since zero is included in the confidence interval of the difference, $95\%$ CI $[-0.19; 0.25]$, see Figure \[fig:replicationComparison\]c and \[fig:replicationComparison\]d. Note that this approach deviated from our appropriate analysis in that it does not compute the median for each individual participant but uses a grand median across participants because the published histogram pools all participants’ RT data. This approach ignores between-subject variance leading to a slight underestimation of the indirect task’s sensitivity. Nevertheless, this additional reanalysis provides converging evidence complementing our reanalysis method.

Comparing the results of our reanalysis of the original study (Figure \[fig:replicationComparison\]b and \[fig:replicationComparison\]c) with the results of our replication experiment (Figure \[fig:replicationComparison\]a) shows that the results are surprisingly consistent. Both studies arrive at very similar estimates for the sensitivities. This corroborates the validity of our reanalysis approach as well as of our replication experiment (see Appendix \[app:validation\] for further validation of our reanalysis approach).

Let us now come back to the potential caveat mentioned above: The congruency effect we found in our replication experiment was smaller than that of the original study (12 ms vs. 24 ms, respectively). Although it would have made our data even more consistent with the original study to find exactly the same congruency effects, this difference does not necessarily mean that there was a larger sensitivity in the original study: Comparing the sensitivity of the indirect tasks in Figures \[fig:replicationComparison\]a and \[fig:replicationComparison\]b indicates that the sensitivities were in fact quite comparable. This can be explained by the trial-by-trial variability also being somewhat larger in the original study, thereby counteracting the larger RT effect. Further research should clarify what exactly are the differences between the original study and our replication that caused these somewhat different congruency effects (while the other results are very consistent). One way to achieve that would be a systematic variation of stimulus parameters, such that the relationship of sensitivities in direct and indirect tasks can be compared across a wide range of sensitivities. Consequently, we are currently determining the role of an [ITA]{} in the particular setting of in a more extensive study, see . To summarize, our reanalysis of as well as our replication of the behavioral responses both suggest that there is no [ITA]{} in the behavioral part of that study. This demonstrates the fundamental flaw of the standard reasoning and suggests that similar problems might exist in other studies.

### Reanalysis of all 15 studies {#reanalysis-of-all-15-studies .unnumbered}

We now apply our reanalysis in a similar way to all other studies. For this, we present the data in a more compact fashion in Figure \[fig:reanalysis\]. For example, what we showed in Figures \[fig:replicationComparison\]b and \[fig:replicationComparison\]d for the study of now corresponds to the lines 7 and 8 in Figure \[fig:reanalysis\], showing the sensitivities for each task in Figure \[fig:reanalysis\]a and the difference of sensitivities in Figure \[fig:reanalysis\]b.

Inspecting Figure \[fig:reanalysis\] shows that there is no consistent evidence for an [ITA]{} in any of the reanalyzed studies. The 15 studies often report multiple experiments, conditions or indirect measures, such that we investigated 44 claimed [ITAs]{} in total. Not a single study showed [ITAs]{} across all conditions, albeit all studies claimed ITAs for all conditions.

Our reanalysis found significant [ITAs]{} in only eight instances, which are spread across five different studies [@Finkbeiner_Palermo_09; @kunde2003conscious; @Naccache_etal_02; @sumner2008mask; @wang2017role]. Note that for multiple hypothesis testing, one would expect at least some false positive results. These results are intermixed with 35 inconclusive results and even an opposite result where the direct task showed significantly larger sensitivity than the indirect task [@Naccache_Dehaene_01_novelStimuli].

Let us stress that our goal was not to investigate whether there exists a “general” [ITA]{} across all studies with their vastly different stimuli, experimental setups, tasks and scientific questions. Therefore, we did not perform a meta-analysis or corrected for multiple testing. This had several reasons. First, our reanalysis favored finding an [ITA]{} by using assumptions that lead to an overestimation of the indirect task sensitivities. We are therefore liberal in confirming [ITA]{} results in each individual study, which makes our arguments stronger if we nevertheless do not find an [ITA]{} (Box \[box:conservativeAnalysis\]). Second, there are additional methodological issues in the reanalyzed studies that introduce further biases, and for which we do not correct in our reanalysis (Box \[box:additionalProblems\]). Considering these biases towards finding an [ITA]{}, a meta-analysis could misleadingly produce the impression that there is a general [ITA]{} in all reanalyzed studies. An [ITA]{} might exist but perhaps only for particular stimuli and setups. Given that the evidence for an [ITA]{} in each individual study is now in question, the research goal should be to differentiate under which conditions a reliable [ITA]{} can be obtained and under which conditions this is not possible. A meta-analysis would not serve this differentiating purpose.

In summary, reanalyzing the results from studies on unconscious priming shows that there is little to no evidence for [ITAs]{}, despite all studies claimed [ITAs]{} for all conditions. Sensitivity in the indirect task is not consistently better than sensitivity in the direct task, as one would expect given that we reanalyzed only those studies, experiments, and conditions for which inference on unconscious processing are based on [ITAs]{}. This demonstrates how seriously the literature on unconscious priming is affected by the flaws of the standard reasoning.

General discussion {#general-discussion .unnumbered}
==================

Many studies in many realms of consciousness research investigating a wide range of cognitive functions are based on the flawed standard reasoning. The main fallacy occurs when the standard reasoning infers an [ITA]{}. That is, a larger sensitivity for masked stimuli in the indirect task as compared to the direct task. In an earlier reanalysis of by , in our replication of the behavioral part of , and in our reanalysis of 15 highly influential studies, we found that there is no study that can consistently claim evidence for an [ITA]{}. In contrast to many claims, responses in the indirect task often seem to show similar sensitivity as in the direct task. In all these cases, it seems sensible to conclude: To the degree to which indirect measures are affected by a masked stimulus, participants can also report the masked stimulus’s identity.

These results question the validity of many claims on unconscious processing. Because an [ITA]{} is a prerequisite for all further-reaching interpretations of the standard reasoning about conscious vs. unconscious processing, our critique affects all these inferences. In particular, the interpretation that masked stimuli were processed to a greater extend than what participants can consciously report and that, thus, processing must have been unconscious is cast in doubt because sensitivity in the indirect task did not prove to be greater than sensitivity in the direct task. Therefore, the fallacy of the standard reasoning has serious consequences for the trustworthiness of the scientific literature on consciousness.

Nevertheless, our results do not necessarily rule out the possibility that [ITAs]{} exist in some cases depending on the particular setup and research question posed. For example, the setting of using color stimuli seems to produce reliable [ITA]{} results because the appropriate analysis was used to compare $d'$ values between the two tasks. Another example applying the appropriate analysis is the study of in which participants were likely to find a previously seen geometric shape attractive but were unable to determine which shapes they have seen before. Therefore, using the indirect attractiveness task produced a higher %-correct when identifying which shapes were previously presented compared to the direct recognition task—an [ITA]{} demonstrated by testing for a difference in %-correct. Therefore, we do not claim that there were never [ITAs]{} and that there were never instances of better sensitivity in the indirect task than in the direct task (and consequently maybe superior and independent unconscious processing). Such a claim would be far beyond the scope of a single scientific study. But we do claim that one of the most prevalent methods in the wide research area of unconscious priming is fundamentally flawed and that this flaw affects and potentially invalidates many studies.

What we suggest is a research program: Given the tremendous interest in unconscious priming and the far-reaching inferences based on studies using the standard reasoning, researchers should reinvestigate the most relevant cases of [ITAs]{} and clarify to which degree the claims in those studies are truly warranted.

In short, the literature needs a serious and concerted reassessment that would go well beyond the scope of a single study and will also require—in critical cases—the collection of new data. In many cases where superior unconscious processing already seemed an established fact, we expect that this finding needs to be revised. In other cases, researchers might still be able to establish such a relationship—which will then be even more interesting and foster the theoretical understanding of when exactly conscious processing is vital for a cognitive function and when it is not.

Author Contributions
====================

S. Meyen, V. H. Franz, and U. von Luxburg developed the methods. I. A. Zerweck conducted the experiment. All authors wrote the manuscript.

Conflicts of interest
=====================

The authors declared no conflicts of interest with respect to the authorship or the publication of this article.

Supplemental Material
=====================

Barnett, AG., Van Der Pols, JC.  Dobson, AJ.

. . .

Bishop, CM.

. . .

Boring, EG.

. . .

Buchner, A.  Wippich, W.

. . .

Cohen, J.

. . .

Cumming, G.

. . .

Custers, R.  Aarts, H.

. . .

Damian, MF.

. . .

Dehaene, S., Lau, H.  Kouider, S.

. .

Dehaene, S., Naccache, L., Cohen, L., Bihan, DL., Mangin, JF., Poline, JB.  Riviere, D.

. . .

Dehaene, S., Naccache, L., [Le Clec’H]{}, G., Koechlin, E., Mueller, M., Dehaene-Lambertz, G., D.

. . .

Dell’Acqua, R.  Grainger, J.

. . .

Dienes, Z.

. .

Dijksterhuis, A., Bos, MW., Nordgren, LF.  Van Baaren, RB.

. . .

[Dulaney]{}, DE.  Eriksen, CW.

. . .

Erdelyi, MH.

. . .

Eriksen, CW.

. . .

Finkbeiner, M.

. . .

Finkbeiner, M.  Palermo, R.

. . .

Franz, VH.  Gegenfurtner, KR.

. . .

Franz, VH. , U.

. . .

Green, DM.  Swets, JA.

. . .

Hassin, RR.

. . .

Hedges, LV.

. . .

Holender, D.

. . .

Janssen, PA., Thiessen, P., Klein, MC., Whitfield, MF., MacNab, YC.  Cullis-Kuhl, SC.

. . .

Jensen, AR.

. . .

Kiefer, M.

. . .

Klauer, KC., Greenwald, AG.  Draine, SC.

. . .

Klotz, W.  Neumann, O.

. . .

Kouider, S.  Dehaene, S.

. . .

Kunde, W., Kiesel, A.  Hoffmann, J.

. . .

Kunst-Wilson, WR.  Zajonc, RB.

. . .

Macmillan, NA.  Creelman, CD.

.  ( ). .

Mattler, U.

. . .

Mattler, U.

. . .

Maxwell, SE.  Delaney, HD.

. . .

Merikle, PM.  Reingold, EM.

. . .

Michel, M., Beck, D., Block, N., Blumenfeld, H., Brown, R., Carmel, D.Yoshida, M.

. . .

Miller, J.

. . .

Miller, J.  Ulrich, R.

. . .

Morris, JS., [Ö]{}hman, A.  Dolan, RJ.

. . .

Morris, JS., [Ö]{}hman, A.  Dolan, RJ.

. . .

Morrison, D.  Henkel, R.

 (). . . .

Naccache, L., Blandin, E.  Dehaene, S.

. . .

Naccache, L.  Dehaene, S.

. . .

Nieuwenhuis, S., Forstmann, BU.  Wagenmakers, EJ.

. . .

Palmer, EM., Horowitz, TS., Torralba, A.  Wolfe, JM.

. .

Peremen, Z.  Lamy, D.

. . .

Pessiglione, M., Schmidt, L., Draganski, B., Kalisch, R., Lau, H., Dolan, RJ.  Frith, CD.

. . .

Rausch, M., Hellmann, S.  Zehetleitner, M.

. . .

Reingold, EM.  Merikle, PM.

. . .

Ribeiro, MJ., Paiva, JS.  Castelo-Branco, M.

. . .

Rouder, JN.  Haaf, JM.

. . .

Schmidt, T.

. . .

Schmidt, T.

. . .

Schmidt, T.  Vorberg, D.

. . .

Shanks, DR.

. . .

Sklar, AY., Levy, N., Goldstein, A., Mandel, R., Maril, A.  Hassin, RR.

. . .

Stahl, J., Gibbons, H.  Miller, J.

. . .

Sumner, P.

. .

[ten Brinke]{}, L., Stimson, D.  Carney, DR.

. . .

[ten Brinke]{}, L., Vohs, KD.  Carney, DR.

. .

Ulrich, R.  Miller, J.

. . .

Vadillo, MA., Linssen, D., Orgaz, C., Parsons, S.  Shanks, DR.

. .

[van den Bussche]{}, E., [van den Noortgate]{}, W.  Reynvoet, B.

. . .

van Gaal, S., Ridderinkhof, KR., Scholte, HS.  Lamme, VA.

. . .

Wang, Y., Wang, Y., Liu, P., Di, M., Gong, Y.  Tan, M.

. . .

Wicherts, JM., Borsboom, D., Kats, J.  Molenaar, D.

. .

Winer, BJ., Brown, DR.  Michels, KM.

.  ( ). .

W[ó]{}jcik, MJ., Nowicka, MM., Bola, M.  Nowicka, A.

. . .

Zehetleitner, M.  Rausch, M.

. . .

Zerweck, IA., Kao, CS., Meyen, S., Amado, C., [von Eltz]{}, M., Klimm, M.  Franz, VH.

. . .

**[Overview appendices]{}** {#app:overview}
===========================

Appendix A: [Overview appendices]{}\
Appendix B: [Validation of reanalysis method]{}\
Appendix C: [Optimality of median classifier]{}\
Appendix D: [Estimating sensitivities from typically reported results]{}\
Appendix E: [Estimating the ratio $q^2$ of between- vs. within-subject variance]{}\
Appendix F: [Details on reanalyzed studies]{}\
Appendix G: [Glossary]{}\

**[Validation of reanalysis method]{}** {#app:validation}
=======================================

We demonstrate in two studies that the appropriate analysis based on the full trial-by-trial data is well approximated by our reanalysis based only on the typically reported statistics (e.g., a $t$ value for the congruency effect in the indirect task). We applied the appropriate analyses using the full, trial-by-trial data and our reanalysis using only the reported summary statistics to the original data from and to our replication based on the study of . Results from the two analyses are very comparable indicating that our reanalysis closely matched the appropriate analysis. In the rest of this section, we present additional evidence for the validity of our reanalysis by using simulated data.

Simulations
-----------

We conducted multiple simulations to validate that our reanalysis method appropriately controls for statistical errors (type I and type II). Each simulation was repeated $10,000$ times. In each run, we generated a trial-by-trial data set with a direct and an indirect task according to the standard repeated measures model outlined in Appendix \[app:preliminaries\]. We simulated $N$ participants with sensitivities, $d'_{\text{true},i}$, independently and randomly drawn from normal distributions with expected value $d'_{\text{true}}$ and standard deviation $q$ (see Appendix \[app:roleOfq\] for why $q$ is the standard deviation of individual true sensitivities). Note that we sampled $d'_{\text{true},i}$ for each participant independently in the direct and indirect task. Applying Signal Detection Theory, each of these individual sensitivities implies two normal distributions separated by $d'_{\text{true},i}$ standard deviations. From these normal distributions, we sampled $M$ responses per condition for a total for $2M$ trials for each participant. We did this twice, once for each task. In the direct task, we compared each response to the true median: If the response lied on the same side as the normal distribution it was sampled from, the simulated binary decision by the participant in this trial was correct, otherwise it was wrong. In the indirect task, we simply treated the drawn responses as the indirect measures (e.g., RTs). We then applied the traditional analysis, the appropriate analysis and our reanalysis. We obtained similar results with log-normal distributions and only report normal distribution results for brevity.

In each simulation, we varied $N$, $M$, $d'_{\text{true}}$ and $q$. If not declared otherwise, the same $q$ was used for data simulation and reanalysis. Only in simulations 5 and 6, we varied the true $q$ with which the data was simulated and used a different $q$ for our reanalysis in order to see how getting this parameter wrong would affect our results.

Simulations 1-3 demonstrate that the standard reasoning miserably fails when applied to the study of . Simulation 4 shows that our replication has sufficient statistical power to find an [ITA]{} if it was there. Simulations 5 and 6 show how our reanalysis would be affected, if the true $q$ was different than what we assumed. We then summarize additional 108 simulations showing that our estimators, even though they use simplifying approximations, are approximately unbiased.

### Simulation 1: Controlling type I errors {#sec:underpower}

We used the same number of participants in the direct ($N = 7$) vs. indirect ($N=12$) task as well as the same number of trials per condition (direct $M=56$ vs. indirect $M=256$) as the original study of . Assuming no [ITA]{}, we set sensitivities in both tasks to be equal (direct $d'_\text{true} = 0.25$ vs. indirect $d'_\text{true} = 0.25$). We assumed $q = 0.15$ for this simulation.

Even though the same sensitivity underlies both tasks, the direct task fails to reach significance half of the time ($51.2\%$) while the indirect task is almost always significant ($99.5\%$). This is not surprising and shows how seriously underpowered the direct task was due to fewer samples, $N$ and $M$. When applying the standard reasoning, a scientist would erroneously conclude an [ITA]{} from a non-significant direct task result and a significant indirect task effect in $48.6$% of experiments. In other words: The widely used standard reasoning would infer an [ITA]{} half of the time even though there is no [ITA]{} present!

Since there is no [ITA]{} present, our reanalysis should find an [ITA]{} only as often as prespecified by the significance level $\alpha=5\%$. Indeed, we find a difference between the two tasks only in 4.7% of the runs. This demonstrates that our reanalysis approach controls appropriately for type I errors.

### Simulation 2: Controlling for type II errors with an underpowered direct task {#app:sim2}

We use the same assumptions as in Simulation 1 except that we now assume there exists an [ITA]{} (direct $d'_\text{true} = 0$ vs. indirect $d'_\text{true} = 0.25$). Since there is an [ITA]{} present, a high statistical power is desired to detect it and avoid type II errors. Typically, a power above $1-\beta = 80\%$ is desired. However, our reanalysis found the [ITA]{} in only $46.2\%$ of the runs. Using the full trial-by-trial data to test for a difference (instead of only using the reported $t$ value from the indirect task) also produced a test power of only $45.9\%$. There is simply not enough data in the direct task to provide sufficient evidence for an [ITA]{}. The problem with lacking statistical power is not located in our reanalysis because also the appropriate analysis has a low statistical power. Instead, the problem is the low sample size in the direct task.

### Simulation 3: Controlling for type II errors with sufficient samples in the direct task {#app:sim3}

We repeated Simulation 2 but increased the number of participants and trials in the direct task to match the ones of the indirect task ($N=12$ and $M=256$). This is most sensible when testing for a difference because a balanced design maximizes statistical power. Here, our reanalysis method detects the [ITA]{} in 78.3% of the runs, which is close to the desired 80%. Using the full trial-by-trial data provides a power of 84.2%. This demonstrates that our reanalysis method provides sufficient power given sufficient samples.

### Simulation 4: Statistical power in our replication {#app:sim4}

We repeated Simulation 3 but used the same number of participants and samples as in our replication study, $N=24$ and $M=128$ in both tasks. There, we have the same amount of observations as (double the participants, half the trials). Here, our reanalysis detects the [ITA]{} in 96.5% of the runs. The appropriate analysis using full trial-by-trial data instead of only a $t$ value in the indirect task achieves 97.0%. The increase in statistical power compared to Simulation 3 comes from sampling more participants which is more efficient than sampling more trials given a fixed total number of observations [@rouder2018power].

### Simulation 5: Overestimating parameter  {#app:sim5}

We repeated Simulation 3, the balanced setting with an [ITA]{}, but generated the data with $q=0.1$. We still use $q=0.15$ for the reanalysis, thus, we overestimate the true $q$. Our reanalysis now successfully detects the [ITA]{} in 99.6% of the runs and so does the appropriate analysis with 99.2%. We detect more [ITAs]{} here than in Simulation 3 because individual sensitivities vary less (lower $q$) so that observed values are more precise. This benefits the appropriate analysis and our reanalysis alike.

### Simulation 6: Underestimating parameter  {#app:sim6}

Repeating Simulation 5, we now simulated the data with $q=0.3$ and kept the parameter of our reanalysis at $q=0.15$, that is, we now underestimate the true $q$. Individual sensitivities vary a lot now. Even though the mean true direct task sensitivity is $d'_\text{true} = 0$ (50%-correct), due to a large standard deviation of $q = 0.3$, 95% of participants’ true sensitivities range between -0.6 (38%-correct) and 0.6 (62%-correct). The assumption $q=0.3$ poses a problem from a theoretical perspective because some participants can discriminate the masked stimuli relatively well (above 60%-correct). But, from a technical perspective, it also decreases the statistical power. With such a large variability between participants, our reanalysis detects an [ITA]{} in only 62.2% of the runs. The appropriate analysis achieves a similarly low power with 69.2%.

### Additional Simulations

We conducted additional simulations, one for each combination of the following parameters: $N\in\{5, 10, 20\}$, $M\in\{50, 100, 200\}$, $d'_\text{true} = \{0, 0.1, 0.2, 0.5\}$, and $q\in\{0.1, 0.15, 0.3\}$. In all these simulations, the average, absolute deviation between true sensitivities, $d'_\text{true}$, and our estimates, $d'_\text{estimated}$ was $\leq 0.01$. A deviation of $0.01$ in terms of sensitivity translates into a deviation as small as 0.2%-correct, which can be considered negligible in this setting—and deviations in simulations with $N\geq 10$ are substantially less.

We computed the standard deviation of $d'_\text{estimated}$ across the 10,000 simulations of each parameter combination. We compared this with the estimated standard error, $SE$. For this purpose, we squared $SE$ of each run, averaged the values and took the square root of the average. For the direct task, the difference between actual variability and our estimates was again $\leq 0.01$. For the indirect task, the same was true when $N\geq 10$. However, for very small sample sizes ($N=5$) our reanalysis deviated to some degree but the absolute difference between actual standard deviation and our estimates still was $\leq 0.05$. Since all reanalyzed studies use sample sizes of $N\geq 10$ in the indirect task, our reanalysis produced approximately unbiased estimates. Overall, our reanalysis approximates the appropriate analysis sufficiently well in the context we applied it to.

**[Optimality of median classifier]{}** {#app:medianClassifier}
=======================================

In the appropriate analysis to infer an [ITA]{}, one needs to transform continuous measurements of the indirect task (e.g., RTs) into a binary classification response. In this step it is important to use the best possible classifier, in order to achieve the highest $d'$ or %-correct values and thereby increase the chance to establish an [ITA]{}. Depending on the type of measurement that is taken in the indirect task (e.g., RT, brain activity, grip force, etc.), this best classifier can have different forms. In many cases, the median classifier is a suitable choice. For example with RT data (as in our replication based on ), the classifier computes for each participant the median RT across all trials and classifies a trial as congruent if the RT is faster than the median and as incongruent if the RT is slower. Below, we prove that the median classifier is optimal in this setting. The proof requires two assumptions:

1.  The indirect measure follows a normal or log-normal distribution with an additive shift between congruent and incongruent conditions. In our case, this assumption is justified because it is well known that RT distributions are well approximated by log-normal distributions [@Ulrich_Miller_93; @palmer2011shapes].

2.  An equal number of observations need to be drawn in both conditions, which is satisfied by the typical experimental design.

Note, that also applied nonparametric machine learning classifiers but—as expected—these approaches could not outperform the median classifier in this scenario.

Classification and statistical optimality in more detail {#OptimalClassification}
--------------------------------------------------------

In this section, we describe how an optimal classification of the single-trial data in an indirect task can be performed. We prove below that, under the assumption that the RTs follow a normal distribution or a log-normal distribution [@Ulrich_Miller_93], the statistically optimal classifier is given by a median split of the reaction times (“median classifier”).

#### General form of the optimal classifier. {#general-form-of-the-optimal-classifier. .unnumbered}

Consider a classification task where the input is a real-valued number $x$ (e.g., a reaction time, RT), and the classifier is supposed to predict one of two labels $y$ (e.g., ’congruent’ or ’incongruent’; for simplicity we use labels 1 and 2 in the following). Following the standard setup in statistical decision theory [@bishop2006pattern section 1.5] we assume that the input data $X$ and the output data $Y$ are drawn according to some fixed (but unknown) probability distribution $P$. This distribution can be described uniquely by the class-conditional distributions $P( X {\, | \,}Y = 1)$ and $P(X {\, | \,}Y = 2)$ and the class priors $\pi_1 = P(Y = 1)$ and $\pi_2 = P(Y=2)$. A classifier is a function $f:{I\!\!R}\to \{1,2\}$ that assigns a label $y$ to each input $x$. The classifier that has the smallest probability of error is called the Bayes classifier. In case the classes have equal weight, that is $\pi_1 = \pi_2$, the Bayes classifier has a particularly simple form: It classifies an input point $x$ by the class that has the higher class-conditional density at this point. Formally, this classifier is given by

\[eq-fopt\] f\_[opt]{}(x) :=

1 & P(X = x [ | ]{}Y =1 ) &gt; P(X = x [ | ]{}Y=2)\
2 &

#### Optimal classifier for normal and log-normal distributions. {#optimal-classifier-for-normal-and-log-normal-distributions. .unnumbered}

We now consider the special case where the class-conditionals follow a particular distribution. Let us start with the normally distributed case. We assume that both class-conditionals are normal distributions with means $\mu_1$, $\mu_2$ and equal variance $\sigma^2$, and we denote their corresponding probability density functions (PDFs) by $\varphi_{\mu_1,\sigma}$ and $\varphi_{\mu_2,\sigma}$. Under the additional assumption that both classes have equal weights $\pi_1 = \pi_2 = 0.5$, the cumulative distribution function (CDF) of the input (marginal distribution of $X$) is given as \[eq-gaussian\] &(x) := 0.5 ( () + ( )), where $\Phi$ denotes the CDF of the standard normal distribution. For $t \in {I\!\!R}$, we introduce the step function classifier with threshold $t$ by \[eq-step\] f\_t(x) :=

1 & x t\
2 &

In the special case where the threshold $t$ coincides with the median of the marginal distribution of $X$, we call the resulting step function classifier the [*median classifier.* ]{}

If the input distribution is given by Equation , then the optimal classifier $f_{opt}$ coincides with the median classifier.

[*Proof.*]{} Because both classes have the same weight of 0.5, the Bayes classifier is given by $f_{opt}$ as in Equation . For any choice of $\mu_1$, $\mu_2$ and $\sigma$, the class-conditional PDFs $\varphi_{\mu_1, \sigma}$ and $\varphi_{\mu_2,\sigma}$ intersect exactly once, namely at $t^* = (\mu_1 + \mu_2) / 2$. By definition of $f_{opt}$, the optimal classifier $f_{opt}$ is then the step function classifier with threshold $t^*$. We now compute the value of the CDF at $t^*$:

(t\^\*) & = 0.5 ( () + ( ))\
&= 0.5 ( () + ())\
&= 0.5 (() + (1 - ())\
& = 0.5. Here, the second last equality comes from the fact that the normal distribution is symmetric about 0. This calculation shows that the optimal threshold $t^*$ indeed coincides with the median of the input distribution, which is what we wanted to prove. $\Box$

It is easy to see that this proof can be generalized to more general types of symmetric probability distributions. It is, however, even possible to prove an analogous statement for log-normal distributions, which are not symmetric themselves. We introduce the notation $\lambda_{\mu,\sigma}$ for the PDF of a log-normal distribution, and $\Lambda_{\mu,\sigma}$ for the corresponding CDF. These functions are defined as &\_[,]{}(x) := (- ) && &&\_[,]{}(x) := ( ).

Consider the case where the class-conditional distributions are log-normal distributions with same scale parameter $\sigma$ but different location parameters $\mu_1$ and $\mu_2$, and assume that both classes have the same weights $\pi_1 = \pi_2 =
0.5$. Then the PDF and CDF of the input distribution (marginal distribution of $X$) are given as & g(x) = 0.5 ( \_[\_1, ]{}(x) + \_[\_2, ]{}(x) )\
& G(x) = 0.5 ( \_[\_1, ]{}(x) + \_[\_2, ]{}(x) ). \[eq-log-normal\]

If the input distribution is given by Equation , then the optimal classifier $f_{opt}$ coincides with the median classifier.

[*Proof.*]{} The proof is analogous to the previous one. For any choice of $\mu_1$, $\mu_2$ and $\sigma$, the densities $\lambda_{\mu_1, \sigma}$ and $\lambda_{\mu_2, \sigma}$ intersect exactly once. To see this, we solve the equation $\lambda_{\mu_1,
  \sigma}(t^*) = \lambda_{\mu_2, \sigma}(t^*)$, which leads to the unique solution $t^*= \exp( (\mu_1 + \mu_2)
/ 2)$. The input CDF at this value can be computed as G(t\^\*) & = 0.5 ( \_[\_1, ]{}(t\^\*) + \_[\_2, ]{}(t\^\*) )\
& = 0.5( ( )+ ( ) )\
& = 0.5. The last step follows as above by the symmetry of the normal cdf. $\Box$

**[Estimating sensitivities from typically reported results]{}** {#app:reanalysisFormulas}
================================================================

We use typically reported results from studies on unconscious priming to estimate the direct and indirect task sensitivities, $d'_\text{estimated,direct}$ and $d'_\text{estimated,indirect}$. First, we recap the basic model assumptions of a standard repeated measures ANOVA and introduce notation. We then derive estimators for the sensitivity and standard error in both tasks using only the typically reported results. Finally, we compute the difference between direct vs. indirect task sensitivities and construct a confidence interval around that difference in order to test for an [ITA]{}.

**Model assumptions** {#app:preliminaries}
---------------------

Our reanalysis of both tasks is based on the standard model of repeated measures ANOVA and paired $t$ test [@Winer_etal_91; @Maxwell_Delaney_00; @rouder2018power] as employed in all reanalyzed studies. In this model $N$ participants perform $M$ trials in each condition. In the specific setting we consider, there are only $2$ conditions. In the direct task, this corresponds to trials where the masked stimulus is from either of two categories, A vs. B. In the indirect task, the two conditions are typically congruent (A-A, B-B) vs. incongruent (A-B, B-A). In each trial, $Y_{ijk}$ denotes the response from participant $i$ ($1,...,N$) in condition $j$ ($1$ or $2$) in trial $k$ ($1,...,M$). In the direct task, responses $Y_{ijk}$ represent participants’ internal evidence of the masked stimuli (some neural activity indicating whether the participant saw a masked stimulus from category A or from B). Based on this noisy internal evidence, participants make an internal classification and guess in each trial to which category the stimulus belonged. In the indirect task, responses $Y_{ijk}$ are the indirect measures (e.g., RTs).

The standard model decomposes participants’ responses $Y_{ijk}$ into five components: $$Y_{ijk} = \mu + p_i + c_j + (p\times c)_{ij} + \epsilon_{ijk}.$$

To facilitate understanding, we now describe the model for the example of congruency effects on RTs in the indirect task; but the same notation applies to other indirect measures and to the direct task as well. RTs have a grand mean $\mu$. Some participants have faster RTs than others which is captured in participants’ effects $p_i$. The congruency condition has an effect $c_j$ on RTs. While $c_1$ is negative leading to faster RTs in congruent trials, $c_2$ is positive reflecting slower RTs in the incongruent conditions. Participants differ in the extent to which the congruency conditions affect them captured in $(p\times c)_{ij}$ so that some participants have a larger congruency effect than others. The variability in the individual effects is captured by this term’s variance, $\operatorname{Var}[(p\times c)_{ij}] = \sigma^2_{p\times c}$. Additionally, there is trial-by-trial noise $\epsilon_{ijk}$ from neuromuscular noise and measurement error leading to different responses in each trial. This trial-by-trial measurement error is assumed by the standard models to have a constant variance (homogeneity) over participants and conditions, $\operatorname{Var}[\epsilon_{ijk}] = \sigma^2_{\epsilon}$. The congruency effect $c_j$ is a fixed effect while participant and interaction effects ($p_i$ and $(p\times c)_{ij}$) are random effects because they depend on the drawn sample of participants. Random effects and trial-by-trial noise are assumed to be normally distributed with an expected value of zero and their corresponding variance.

### Raw effects and sensitivities {#raw-effects-and-sensitivities .unnumbered}

Each participant $i$ has an individual expected congruency effect, $\Delta_i$, which theoretically would be obtained by sampling infinite trials. The expected RT difference across participants is denoted by $\Delta$. $$\begin{aligned}
        \Delta_i &= \left(c_2 + (p\times c)_{i2}\right) - \left(c_1 + (p\times c)_{i1}\right)\\
                \Delta &= c_2-c_1\end{aligned}$$ In a typical experiment, the individual congruency effects are estimated by the observed mean difference between conditions. For the $i$-th participant, this estimate is $\hat{\Delta}_i$ and averaged across participants this is $\hat{\Delta}$. $$\begin{aligned}
        \hat{\Delta}_i &= \bar{Y}_{i2\cdot} - \bar{Y}_{i1\cdot} = \frac{1}
        {M}\sum_{k=1}^M Y_{i2k} - \frac{1} {M}\sum_{k=1}^M Y_{i1k}\\
                \hat{\Delta} &= \frac{1}{N} \sum_{i=1}^N \hat{\Delta}_i\end{aligned}$$

A participant’s true sensitivity $d'_{\text{true},i}$ is the normalized effect—normalized by the trial-by-trial error standard deviation $\sigma_\epsilon$. This quantity indicates, similar to a signal to noise ratio, how well a participant’s RTs are separable and therefore to which degree the masked stimuli were processed, cf. Figure \[fig:toyExample\]a. The expectation across participants is the true sensitivity $d'_\text{true}$ indicating how well the RTs of a prototypical participant are separated. $$\begin{aligned}
        d'_{\text{true},i} & = \frac{\Delta_i}{\sigma_\epsilon}\\
                d'_\text{true} &= \frac{\Delta}{\sigma_{\epsilon}}.\end{aligned}$$

In the direct task, $d'_\text{true}$ is typically measured by the sensitivity index $d'$ averaged across participants. Participants’ individual $d'_i$ are calculated from hit rate, $\text{HR}$ (%-correct guesses for masked stimuli from category A), and false alarm rate, $\text{FA}$ (%-incorrect guesses for masked stimuli from category B), where $\Phi^{-1}$ is the inverse cumulative density function of the normal distribution. $$\begin{aligned}
    d'_i &= \Phi^{-1}(\text{HR}) - \Phi^{-1}(\text{FA})\\
        d' &= \frac{1}{N}\sum_i d'_i.\end{aligned}$$ Note that the empirical literature often uses the notation $d'$ without a clear distinction between estimated vs. true value and individual vs. average effects. Because we need to be more precise in our derivations: We denote the true value of an individual participant by $d'_{\text{true},i}$ and the sensitivity index, which is an estimate for the true value, by $d'_i$. We denote the true sensitivity across participants by $d'_\text{true}$. In the direct task, this is estimated by the average across $d'_i$ values denoted by $d'$. We will also label this averaged estimate $d'_\text{estimated,indirect}$. Since the indirect task does not offer such an estimate, we use the reported results to come up with an estimate for the true sensitivity, which we will call $d'_\text{estimated,indirect}$.

### Two variance sources: true effect (between-) vs. trial-by-trial error (within-subject) variance {#two-variance-sources-true-effect-between--vs.-trial-by-trial-error-within-subject-variance .unnumbered}

Participants differ in their true congruency effect. The variance of these true inter-individual differences can be derived from the model variances using the standard assumptions (1) $(p\times c)_{ij} \sim \mathcal{N}(0,\sigma^2_{p\times c})$, (2) $Var[c_1] = Var[c_2] = 0$, and (3) $(p\times c)_{i1} = -(p\times c)_{i2}$. We denote this true effect variance as $\sigma^2_{\text{effect}}$: $$\begin{aligned}
        \sigma^2_{\text{effect}} &= \operatorname{Var}[\Delta_i] = \operatorname{Var}[[c_2 + (p\times c)_{i2}] - [c_1 + (p\times c)_{i1}]]
        \\
        &= \operatorname{Var}[(p\times c)_{i2} - (p\times c)_{i1}] = \operatorname{Var}[2(p\times c)_{i2}]
        \\
        &= 4\sigma^2_{p\times c}.\end{aligned}$$

The variance of the actually observed congruency effects is conceptually different from the variance of the true effects. We denote the variance of the observed congruency effects as $\sigma^2_{\hat{\Delta}_i}$. The observed congruency effects vary more because they are not only affected by true inter-individual difference but also by trial-by-trial measurement errors: $$\begin{aligned}
    \sigma^2_{\hat{\Delta}_i} &= \operatorname{Var}[\hat{\Delta}_i] \\ 
    &= \operatorname{Var}\left[\bar{Y}_{i2\cdot} - \bar{Y}_{i1\cdot}\right]\\ &= \operatorname{Var}\left[\frac{1}{M} \left(\sum_{k=1}^M \mu + p_i + c_2 + (p\times c)_{i2} + \epsilon_{i2k}\right) \right. \\
    & ~~~~~~~~~~~~~~~~~~~ - \left. \frac{1}{M} \left(\sum_{k=1}^M \mu + p_i + c_1 + (p\times c)_{i1} + \epsilon_{i1k}\right)\right] \\ 
    &= \operatorname{Var}\left[[c_2 + (p\times c)_
    {i2}] - [c_1 + (p\times c)_{i1}] + \frac{1}{M} \left(\sum_{k=1}^M \epsilon_
    {i2k}\right) - \frac{1}{M} \left(\sum_{k=1}^M \epsilon_{i1k}\right)\right]\\
    &= \operatorname{Var}\left[\Delta_i +
    \frac{1}{M} \left(\sum_{k=1}^M \epsilon_{i2k}\right) - \frac{1}{M}
    \left(\sum_{k=1}^M \epsilon_{i1k}\right)\right]\\ 
    &= \operatorname{Var}[\Delta_i] + \operatorname{Var}\left[\frac{1}{M} \sum_{k=1}^M \epsilon_{i2k}\right] + \operatorname{Var}\left[\frac{1}{M}
    \sum_{k=1}^M \epsilon_{i1k}\right]\\ 
    &= \sigma^2_{\text{effect}} + \frac{2}{M}\sigma^2_\epsilon.\end{aligned}$$ This has an implication for the variance of average congruency effects, $\hat \Delta = \frac{1}{N}\sum_i \hat \Delta_i$. These observed, average congruency effects vary due to two variance sources, the true inter-individual differences and trial-by-trial measurement error. $$\hat{\Delta} \sim \mathcal{N}\left(\Delta, 
    \frac{\sigma^2_{ \text{effect}} + \frac{2} {M}\sigma^2_\epsilon}{N}
    \right).$$

We will later have to estimate $\sigma^2_\epsilon$ from a given $\sigma^2_{\hat{\Delta}_i}$. To achieve this, we must disentangle $\sigma^2_{\text{effect}}$ and $\sigma^2_\epsilon$. We do so by defining the ratio $q^2$ between these two sources of variance: $$q^2 = \frac{\sigma^2_{\text{effect}}}{\sigma^2_\epsilon}.$$ This parameter tells us how much of the observed variability comes from true differences vs. noise. If $q^2=0$ then all participants would have the same true congruency effect and observed differences are only due to trial-by-trial error. If $q^2$ is large then there is relatively small trial-by-trial error variance and observed differences between participants stem from reliable, true differences between subjects. Crucially, note that $q^2$ is also the variance of true, individual sensitivities. $$\begin{aligned}
     \text{\textit Var}[d'_{\text{true},i}] = Var\left[ \frac{\Delta_i}{\sigma_\epsilon}  \right] 
     = \frac{\sigma^2_\text{effect}}{\sigma^2_\epsilon} 
     = q^2.\end{aligned}$$ Thus, the square root of this ratio, $q$, is the standard deviation of true, individual sensitivities. $$\begin{aligned}
    q = SD[d'_{\text{true},i}]\end{aligned}$$ We derive a reasonable value to use for our setting in Appendix \[app:roleOfq\], which is $q^2=0.0225$. This means that we will assume that participants’ sensitivities $d'_{\text{true},i}$ vary around some true value $d'_\text{true}$ with a standard deviation of $q = 0.15$.

### Relation between sensitivity and accuracy {#relation-between-sensitivity-and-accuracy .unnumbered}

As we have already mentioned, some published studies report $d'$ values, whereas other studies report %-correct values in the direct task. Because we would like to be able to work with either of them, we now discuss the relationship that can transform %-correct values into $d'$ values and vice versa.

Recall that $d'_{\text{true},i}$ denotes the true sensitivity of participant $i$, and let us introduce the notation $\pi_i$ for the true probability of a correct classification of a masked stimuli by participant $i$. We now make the assumption of a neutral criterion in the direct task, that is, participants are not inclined to guess one category of the masked stimuli (A or B) more often than the other. Under this assumption, the true relationship is $d'_{\text{true},i} = 2\Phi^{-1}(\pi_i)$ where $\Phi^{-1}$ is the inverse cumulative normal distribution [@Macmillan_Creelman_90]. To simplify our later analysis, we now introduce the linear approximation $h(x) = 5(x - 0.5) \approx 2\Phi^{-1}(x)$. This approximation works remarkably well in the regime of sensitivities being close to zero: $$\text{given }\pi_i\text{, we approximate }  d'_{\text{true},i} \approx h(\pi_i) = 5(\pi_i-0.5)$$ $$\text{given }d'_{\text{true},i}\text{, we approximate }\pi_i \approx h^{-1}(d'_{\text{true},i}) = \frac{1}{5}d'_{\text{true},i}+0.5$$ For example, an accuracy of 54%-correct is approximately translated into the sensitivity $d'_{\text{true},i} \approx 5\cdot(0.54-0.5) = 0.2$. This is very close to the true translation, $d'_{\text{true},i} = 2\Phi^{-1}(\pi_i) = 0.201$. Table \[tab:approximation\] shows that this approximation provides an almost perfect fit in the range of $\pi_i \in [0.4;0.6]$ or, equivalently, $d'_{\text{true},i} \in [-0.5; 0.5]$. Larger values, that is, an accuracy above 60%-correct, would be at odds with the experimental setting in which direct task performance is assumed to be close to chance ($d'_{\text{true},i}$ close to 0 and $\pi_i$ close to 0.5).

    $\pi_i$   $h(\pi_i)$   $d'_{\text{true},i}$
  --------- ------------ ----------------------
       0.50        0.000                  0.000
       0.52        0.100                  0.100
       0.54        0.200                  0.201
       0.56        0.300                  0.302
       0.58        0.400                  0.404
       0.60        0.500                  0.507
       0.62        0.600                  0.611
       0.64        0.700                  0.717
       0.66        0.800                  0.825
       0.68        0.900                  0.935
       0.70        1.000                  1.049

  : Relation between the true accuracy (first column), the approximation of the sensitivity (second column) and the true sensitivity (third column). Note that for $\pi_i$ in the range of $[0.5,0.6]$ and $D_i$ in the range of $[0,0.5]$ (first six rows in the table) there is an almost perfect fit between $h(\pi_i)$ and $d'_{\text{true},i}$. Negative values of $d'_{\text{true},i}$ follow symmetrically.

\[tab:approximation\]

**Estimated sensitivity, , from mean sensitivity index**  {#app:Dfromsensitivity}
---------------------------------------------------------

We want to estimate the sensitivity and corresponding standard error from the typically reported direct task results. Usually, the average over individual sensitivity indices is reported as $d'$. This sensitivity index is already a measure for the true sensitivity and we take it as it is [@Macmillan_Creelman_90], $$\begin{aligned}
    d'_\text{estimated,direct} = d'. {\addtocounter{equation}{1}\tag{\theequation}}\label{eqn:DfromdprimeUnocrrected}\end{aligned}$$ The standard error of $d'$ is composed of two variances, one due to systematic variation between individuals’ true sensitivities ($d'_\text{true,i}$) and the other due to non-systematic measurement error ($\epsilon_{d'_i}$). We use two simplifications: (a) We neglect dependencies between them because the variance of random error $\operatorname{Var}[\epsilon_{d'_i}]$ does not change substantially for different sensitivity values in the relevant range, $D_i^\text{dir} \in [-0.5, 0.5]$; (b) We apply the approximation function $h$ that relates $d'_i$ to $\hat\pi_i$. This allows us to use the variance of the binomially distributed accuracies $\hat\pi_i$ from $2M$ trials, $\operatorname{Var}[\epsilon_{\hat \pi_i}] = \pi_i(1-\pi_i)/(2M)$, and relate them back to the variance of $d'_i$, which leads to $\operatorname{Var}[\epsilon_{d'_i}] \approx 5^2\operatorname{Var}[\epsilon_{\hat \pi_i}]$. $$\begin{aligned}
        SE_\text{direct} &= \sqrt{\operatorname{Var}\left[d'\right] } \\
        &= \sqrt{\operatorname{Var}\left[\frac{1}{N} \sum_i d'_i\right]} = \frac{1}{\sqrt{N}}\sqrt{\operatorname{Var}[d'_i]} 
        = \frac{1}{\sqrt{N}}\sqrt{\operatorname{Var}[d'_\text{true,i} + \epsilon_{d'_i}]}\\
        &\overset{(a)}{\approx} \frac{1}{\sqrt{N}}\sqrt{\operatorname{Var}[d'_\text{true,i}] + \operatorname{Var}[\epsilon_{d'_i}]}
        \overset{(b)}{\approx} \frac{1}{\sqrt{N}}\sqrt{\operatorname{Var}[d'_\text{true,i}] + 5^2\operatorname{Var}[\epsilon_{\hat \pi_i}]}\\
        & \overset{}{=} \underbrace{\frac{1}{\sqrt{N}}}_\text{average}\sqrt{\underbrace{q^2}_{\text{between subject variation}} + ~5^2~ \underbrace{\frac{\left(\frac{1}{5}d'+0.5\right)\left(1-\left(\frac{1}{5}d'+0.5\right)\right)}{2M}}_\text{non-systematic error of $\hat \pi_i$}}         {\addtocounter{equation}{1}\tag{\theequation}}\label{eqn:SEfromdprime}.\end{aligned}$$ Without simplifications (a) and (b), one could construct an exact estimator. Exact calculations from show that $d'$ slightly overestimates the true sensitivity $d'_\text{true}$ but that this bias is so small that the estimator can considered approximately unbiased when sample sizes as in our context are used. On the other hand, our simplifications allow us to find a closed form solution that is simple to compute. Our estimators are well aligned with the true values, which we have shown by validating simulations in Appendix \[app:validation\].

Estimated sensitivity, , from mean accuracy  {#app:Dfromacc}
--------------------------------------------

Instead of $d'$, some studies report the average classification accuracy $\hat{\pi}$ (%-correct) for the direct task. We estimate the sensitivity $d'_\text{estimated,direct}$ from the mean accuracy $\hat{\pi}$ by a plug-in estimator [@Macmillan_Creelman_90], $$d'_\text{estimated,direct} = 2\Phi^{-1}(\hat{\pi}) \approx 5\cdot(\hat\pi - 0.5)
        {\addtocounter{equation}{1}\tag{\theequation}}\label{eqn:Dfrompi}$$ where $\Phi^{-1}$ is the inverse cumulative normal distribution. Exploiting the linearity of approximation $h$ in (\*), we can derive that this estimator is approximately unbiased: $$\begin{aligned}
 \text{E}[d'_\text{estimated,direct}]
        &= \text{E}[2\Phi^{-1}(\hat{\pi})] \approx \text{E}[h(\hat{\pi})] \overset{(*)}{=} h[\text{E}(\hat{\pi})] = h(\pi) \approx 2\Phi^{-1}(\pi) = d'_\text{true}.\end{aligned}$$ Next, the standard error can be derived in the same fashion as for reported $d'$ values so that we obtain: $$\begin{aligned}
    SE_\text{direct} = \frac{1}{\sqrt{N}}\sqrt{q^2 + 5\frac{\hat 
    \pi \left(1-\hat \pi \right)}{2M}}. {\addtocounter{equation}{1}\tag{\theequation}}\label{eqn:SEfrompi}\end{aligned}$$

Estimated sensitivity, , from and values {#app:Dfromt}
----------------------------------------

Now let us move to estimating sensitivities from $t$ values in the indirect task. We will show that an unbiased estimator is obtained from multiplying the $t$ value by the constant $\kappa_{N,M,q^2}$: $$d'_\text{estimated,indirect} = t\cdot \kappa_{N,M,q^2} ~~~~\text{with}~~~~~ \kappa_{N,M,q^2}
        =
        \frac{\Gamma\left(\frac{N-1}{2}\right)}{
        \sqrt{\frac{N-1}{2}}\Gamma\left( \frac{N-2}{2} \right)}\sqrt{\frac{Mq^2 + 2}{NM}},
        {\addtocounter{equation}{1}\tag{\theequation}}\label{eqn:Dfromt}$$ Where $\Gamma$ is the gamma distribution.

Under our model assumptions, the $t$ value is computed from the observed congruency effect divided by the estimated standard error: $$t = \frac{ \hat{\Delta} }{\hat{\sigma}_{\hat{\Delta}_i} } 
        \sqrt{N}$$ We know that $\hat{\Delta} \sim \mathcal{N}\left(\Delta, (\sigma^2_{ \text{effect}} + \frac{2} {M}\sigma^2_\epsilon)/N \right)$ from above. Now we introduce independent random variables $Z \sim \mathcal{N}\left(0, 1\right)$ and $V \sim \chi^2\left (N-1\right)$ and rewrite the $t$ value as follows: $$\begin{aligned}
        t &= \frac{Z\sqrt{\left(\sigma^2_{\text{effect}} + \frac{2}
{M}\sigma^2_\epsilon\right)/N} + \Delta }
{ \sqrt{ \left(\sigma^2_{\text{effect}} + \frac{2}
{M}\sigma^2_\epsilon\right) }\sqrt{\frac{V}
{N-1}} }\sqrt{N} \\ 
&= \left(Z+\Delta\sqrt{\frac{N}{\left(\sigma^2_{\text{effect}} + \frac{2}
{M}\sigma^2_\epsilon\right)}}\right)\frac{\sqrt{N-1}}{\sqrt{V}}\end{aligned}$$ We now use $\sigma^2_{\text{effect}} = q^2 \sigma^2_\epsilon$ (also from the preliminaries) to isolate $\sigma^2_\epsilon$ and obtain $d'_{true}$. $$\begin{aligned}
        t &= \left(Z+\Delta\sqrt{\frac{N}{\left(q^2\sigma^2_\epsilon + \frac{2} {M}\sigma^2_\epsilon\right)}}\right)\frac{\sqrt{N-1}}{\sqrt{V}} \\
        &= \left(Z+\frac{\Delta}{\sigma_\epsilon}\sqrt{\left(\frac{NM}{Mq^2 + 2}\right)}\right)\frac{\sqrt{N-1}}{\sqrt{V}} \\
        &= \left(Z+d'_{true}\sqrt{\left(\frac{NM}{Mq^2 + 2}\right)}\right)\frac{\sqrt{N-1}}{\sqrt{V}} \\\end{aligned}$$ As a result, $t$ follows a $t$ distribution with degrees of freedom $df = N-1$ and non-centrality parameter $\delta =
d'_{true}\sqrt{\frac{NM}{Mq^2 + 2}}$. We know the expected $t$ value to be $\text{E}[t] = \kappa^{-1}_{N,M,q^2}
\cdot d'_\text{true}$ [@hedges1981distribution]. In consequence, an unbiased estimator of $d'_\text{true}$ is $d'_\text{estimated,indirect} = t\cdot \kappa_{N,M,q^2}$.

As for the expected value, the variance of $t$ values is also given by the properties of a non-central $t$ distribution. Multiplying this variance by the constant $\kappa_
{N,M,q^2}$ yields the variance of our estimated sensitivity $
d'_\text{estimated,indirect}$. Since this depends on the non-centrality parameter, we use the plugin estimator $\hat{\delta} = d'_\text{estimated,indirect}\sqrt{\frac{NM}{Mq^2 + 2}}$. The standard error being its positive square root follows accordingly: $$\begin{aligned}
    SE_\text{direct}
    &= \sqrt{\operatorname{Var}[\kappa_{N,M,q^2} \cdot t]} 
    = \kappa_{N,M,q^2}\sqrt{\operatorname{Var}[t]} \\
    &= \kappa_{N,M,q^2}\sqrt{\frac{1+\hat\delta^2}{N-3} - \frac{\hat\delta^2(N-1)\Gamma\left( \frac{N-2}{2} \right)^2}{2\Gamma\left(\frac{N-1}{2} \right)^2}} \\
    &= \kappa_{N,M,q^2}\sqrt{\frac{1+(d'_\text{estimated,indirect})^2\frac{NM}{Mq^2 + 2}}{N-3} - \frac{(d'_\text{estimated,indirect})^2\frac{NM}{Mq^2 + 2}(N-1)\Gamma\left( \frac{N-2}{2} \right)^2}{2\Gamma\left( \frac{N-1}{2} \right)^2}}\\
    &= \kappa_{N,M,q^2}\sqrt{\left(1+\frac{2t^2}{N-1}\left(\frac{\Gamma(\frac{N-1}{2})}{\Gamma(\frac{N-2}{2})}\right)^2 \right) \left(\frac{N-1}{N-3}\right) - t^2}
    {\addtocounter{equation}{1}\tag{\theequation}}\label{eqn:SEfromt}\end{aligned}$$ With this, we can estimate the sensitivity and its standard error from a give $t$ value in a repeated measures design.\
Finally, note that this approach can be applied identically to reported $F$ values instead of $t$ values. The reason is that in repeated measures ANOVA settings with two conditions the equality $|t| = \sqrt{F}$ holds. The main argument can be derived in the following equations using the standard definitions for the explained (SSE) and residual summed squares (SSR), see :

$$\begin{aligned}
    t^2 &= \left( \frac{ \hat{\Delta} } { \hat{\sigma}_{\hat{\Delta}_i} } \cdot  \sqrt{N} \right)^2 
        = \frac{4 \cdot N \cdot ( \hat{\Delta}/2 )^2 }{  \frac{1}{N-1} \sum_{i=1}^{N} \left(\hat{\Delta}_i - \hat{\Delta} \right)^2  } 
        = \frac{2 \cdot N \cdot ( \hat{\Delta}/2 )^2 }{  \frac{1}{N-1} \sum_{i=1}^{N} \left(\frac{\hat{\Delta}_i - \hat{\Delta}}{2} \right)^2  } \\
        &= \frac{2 \cdot N \cdot ( \hat{\Delta}/2 )^2 }{  2\sum_{i=1}^{N} \left(\frac{\hat{\Delta}_i - \hat{\Delta}}{2} \right)^2/(2N-2)  } = \frac{\text{SSE}/df_\text{E}}{\text{SSR}/df_\text{R}} 
        = F.\end{aligned}$$

Confidence intervals for the difference in sensitivities {#app:ci}
--------------------------------------------------------

Based on the previous estimators, we now need to test for a significant difference between sensitivities in direct vs. indirect tasks. For this purpose we construct a 95% confidence interval around the difference $d'_\text{difference}$ while taking the standard error $SE_\text{difference}$ of the estimated difference into account: $$\begin{aligned}
        d'_\text{difference} &= d'_\text{estimated,indirect} - d'_\text{estimated,direct}  {\addtocounter{equation}{1}\tag{\theequation}}\label{eqn:Ddiff} \\
        SE_\text{difference} &= \sqrt{(SE_\text{direct})^2 + (SE_\text{indirect})^2}  {\addtocounter{equation}{1}\tag{\theequation}}\label{eqn:SEdiff} \\
        95\%\text{~CI} &= 
        \left[d'_\text{difference} \pm z_{0.975} \cdot SE_\text{difference} \right]
        {\addtocounter{equation}{1}\tag{\theequation}}\label{eqn:ci}, \end{aligned}$$

where $z_{0.975}=1.96$ is the 97.5% quantile of the normal distribution. If zero is included in the confidence interval, $0 \in 95\%\text{~CI}$, then there is not sufficient evidence for an [ITA]{} because the observed difference can be explained by measurement error in a situation where the true direct and indirect task sensitivities are equal. Only if the confidence interval lies above zero, that is $95\%\text{~CI}=[a,b]$ and $a>0$, there is evidence for the presence of an [ITA]{}.

Note that in this test we use quantiles $z_\alpha$ of the normal distribution and not quantiles of the $t$ distribution. Using the $t$ distribution would require to estimate the degrees of freedom, which is unnecessarily complicated for our approach. We use the quantiles of the normal distribution which leads to a more liberal test, that is, our approach confirms an [ITA]{} more often than if we were using $t$ quantiles. We are therefore more conservative in our critique and our argument is stronger when we do not confirm an [ITA]{}.

**[Estimating the ratio $q^2$ of between- vs. within-subject variance]{}** {#app:roleOfq}
==========================================================================

As we have seen in the reanalysis of direct and indirect task sensitivities, we need to know one particular parameter: $q^2$, a ratio of two variances. This is not an artifact of our analysis but unavoidable. To see why, consider estimating the indirect task sensitivity $d'_\text{estimated,indirect}$ from $t$ values. A $t$ value is computed, roughly speaking, by dividing an observed effect by its standard error, $t = \bar{x}/SE$. For example, $\bar{x}$ is the average congruency effect and $SE$ is the estimated standard error of congruency effects across participants. This standard error is influenced by two sources of variability: variance due to true, inter-individual difference in congruency effects across participants ($\sigma^2_{\text{effect}}$) and variance due to trial-by-trial measurement error ($\sigma^2_{\epsilon}$). We want to isolate $\sigma^2_{\epsilon}$ in order to estimate the sensitivity $d'_\text{true} = \Delta/\sigma_{\epsilon}$. We do so by defining the ratio of the two sources of variability: $$q^2 = \frac{\sigma^2_{\text{effect}}}{\sigma^2_\epsilon}.$$ We know that this parameter is also equal to the variance of individual true sensitivities. Therefore, it might be more intuitive to think of this value un-squared: $q$ is the standard deviation of participants’ true sensitivities, $q = \text{SD}[d'_{\text{true},i}]$.

We considered multiple studies making explicit assumptions about the ratio of between- vs. within-subject variance as well as our own replication study. All these studies suggest a specific value for $q^2$. Our results are summarized in Table \[qtable\]. For our reanalysis, we use the assumption: $$q^2 = 0.0225 \iff q = 0.15.$$

  ------------------------ ------------------------------------- ------------------- ------------ -----------------------
                             Reanalysis of Dehaene et al. (1998)                                  
  Study                                   $\sigma_\text{effect}$   $\sigma_\epsilon$    **$q^2$**  $d'_\text{estimated}$
  Rouder & Haaf (2018)                                      9.88                 104    **0.009**          0.21
  Our replication                                           6.70                  78    **0.007**          0.20
  Miller & Ulrich (2013)                                   10.78                  92    **0.014**          0.24
  Jensen (2002)                                            10.84                  91    **0.014**          0.25
  Ribeiro et al. (2016)                                    11.55                  79    **0.021**          0.28
  Our assumption                                           11.63                  78   **0.0225**          0.29
  ------------------------ ------------------------------------- ------------------- ------------ -----------------------

  : Based on different studies, we used different assumption for the parameter $q^2$ to conducted our reanalysis of the indirect task sensitivity from (last column). Larger values of $q^2$ increase the estimated, indirect task sensitivity.[]{data-label="qtable"}

Large values of $q$ favors the [ITA]{} hypothesis in our reanalysis because larger values of $q$ attribute more variance to $\sigma^2_{ \text{effect}}$, then $\sigma^2_\epsilon$ is smaller and, in turn, $d'_\text{estimated,indirect}$ is larger. Hence, overestimating $q^2$ means increasing the probability of confirming an [ITA]{}. Not discovering an [ITA]{} under those circumstances makes our argument even stronger. Accordingly, the largest $q^2$ value we found is $0.021$. Thus, as mentioned above, we use $q^2 = 0.0225$ for our reanalysis corresponding to the squared value $q=0.15$. To gauge the effect of this parameter, we also conducted our reanalysis with different values of $q^2$. Specifically, we used a more realistic assumption ($q^2 = 0.01$) in Figure \[fig:reanalysisSmaller\] and an overly unrealistic assumption ($q^2=0.09$) in Figure \[fig:reanalysisLarger\]. The rest of this section describes the individual studies providing evidence for certain $q^2$ values.

discuss the relation between the two sources of variance in psychophysics. Their formulas are identical to ours when changing the notation from $\sigma^2_{\text{effect}}$ to $\sigma^2_\beta$ and $\sigma^2_{\epsilon}$ to $\sigma^2$. They argue that reasonable values in psychophysics settings are $\sigma_{\text{effect}} = 28$ ms and $\sigma_{\epsilon} = 300$, which leads to $q^2 = \sigma^2_{\text{effect}}/\sigma^2_{\epsilon} = 0.009$, (first row in table \[qtable\]).

In accordance with this result, our replication based on also produced the estimated the ratio to be $\hat{q}^2 = (6.7ms / 78ms)^2 = 0.007$.

Other studies did not discuss the ratio between the two variances $\sigma^2_{\text{effect}}$ and $\sigma^2_{\epsilon}$ but only the trial-by-trial error variability $\sigma^2_{\epsilon}$. This alone is not sufficient to determine $q^2$. However, we use these values by combining them with the reported statistics from . This is possible because there is only one degree of freedom: Knowing $q^2$ or $\sigma^2_{\epsilon}$ is interchangeable when reanalyzing one study. Specifically, noted that $\hat{\sigma}^2_{\hat{\Delta}_i} = \hat\sigma^2_{\text{effect}} + \frac{2}{M}\hat\sigma^2_\epsilon = (13.5~ms)^2$ in a setting with $M=256$. To uniquely identify $\hat\sigma^2_{\text{effect}}$ and $\hat\sigma^2_{\epsilon}$ we only need either $\hat{q}^2$ or $\hat\sigma^2_{\epsilon}$. Thus, we first estimate the between-subjects variance, $\hat\sigma^2_{\text{effect}} = (13.5~ms)^2 - \frac{2}{M}\hat\sigma^2_\epsilon$, and then the ratio parameter, $q^2 = \hat\sigma^2_{\text{effect}} / \hat\sigma^2_\epsilon$. This way, we determined $q^2$ from the following three studies and obtained the values shown in Table \[qtable\].

suggested $\sigma_\epsilon = 92$ ms in a binary forced-choice task (without masked stimuli). Combining this with Dehaene et al.’s results as just described yields $\sigma_{\text{effect}} = 10.8$ ms and therefore $q^2 = 0.014$.

report an average estimate of $\hat{\sigma}_\epsilon = 91$ ms measured in $N=863$ nine to twelve year olds yielding $q^2 = 0.014$. report $\hat{\sigma}_\epsilon = 79$ ms in a speeded binary choice task without priming suggesting $q^2 = 0.021$. Even though the specific tasks and populations from these two studies do not match our setting exactly, it is plausible that variances are comparable.

Even though we only discussed behavioral data, the situation is more extreme (smaller $q$) for EEG and fMRI data because single-trial event related potentials (ERPs) or blood-oxygen-level-dependent signals (BOLD signals) incorporate much more noise [@stahl2010modeling]. Thus, the ratio between effect vs. noise variance in these measures will be even smaller justifying our choice of $q^2 = 0.0225$.

**[Details on reanalyzed studies]{}** {#app:reportedResults}
=====================================

For each study, we give an overview of the study’s structure, indicate in a table which values we extracted and explain our decisions for in- and exclusion of particular results. We only used results that follow the standard reasoning and fit into our reanalysis method. We use the following abbreviations:

1.  **N**ot **r**eanalyzable: Reported statistics do not match our reanalysis method. For example when the congruency factor has more than two levels (congruent, incongruent, and neutral) or when there are additional between-subject factors.

2.  **N**o **i**ndirect **e**ffect: The study attempted to find an [ITA]{} but failed due to a non-significant indirect task result. In such cases, the studies usually abort the standard reasoning, such that these cases are not relevant for us.

3.  **R**egression **t**o the **m**ean: The study excluded participants with good direct-task performance (cf. Box \[box:additionalProblems\] of why this is problematic). In these cases we still reanalyzed the reported results. Note that our reanalysis can be expected to confirm an [ITA]{} even when the underlying sensitivities are equal.

4.  **S**tandard **r**easoning: We included quotes from the reanalyzed studies indicating that they adhere to the standard reasoning.

We report the number $N$ of participants, the total number of observed trials, and the reported statistic of the original study. Additionally, we report the sensitivities and standard errors of our reanalysis. These are the values from Figure \[fig:reanalysis\]a. We then report the differences in sensitivities and their standard errors; here the difference is always taken between the current row’s indirect task compared to the previously reported direct task. These results are presented in Figure \[fig:reanalysis\]b. We abbreviate Experiment 1 by E1, etc.

### 

The study reports four experiments but concludes an [ITA]{} only in Experiment 1 and 4. Experiments 2 and 3 were NIE.

SR: “Two control experiments investigated participants’ ability to consciously perceive the masked primes. It was shown that performance was at chance level on both presence-absence judgments and on a number vs. random letter string discrimination task when the temporal characteristics of a trial were identical to those of the main experiment. Thus, the congruity effect described above must indeed have occurred outside of the participants’ awareness” (p. 1).

  --------------------------- ---- --------------- ------------------ ------------------------------ ------------------------------------
                                                                                                     
  (l)[2-4]{} (l)[5-6]{}          N   no. of trials Statistic            $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct (E1)        16              96 $d' = 0.064$                        $0.06\pm0.07$ 
  Indirect, RT (E1)             16             120 $F(1,15) = 6.15$                    $0.14\pm0.07$                        $0.07\pm0.10$
  Indirect, error rate (E1)     16             120 $F(1,15) = 13.8$                    $0.21\pm0.07$                        $0.14\pm0.10$
  (l)[1-6]{} Direct (E4)        16              96 $d' = 0.117$                        $0.12\pm0.07$ 
  Indirect, RT (E4)             16             120 $F(1,15) = 5.67$                    $0.13\pm0.07$                        $0.02\pm0.10$
  Indirect, ER (E4)             16             120 $F(1,15) = 5$                       $0.13\pm0.07$                        $0.01\pm0.10$
  --------------------------- ---- --------------- ------------------ ------------------------------ ------------------------------------

  : Reported results from Damian (JEP:HPP, 2001)

### 

The study reported two direct tasks and three indirect tasks. From the two direct tasks, we consider only the second direct task (word vs. digit discrimination) because it fits the neutral criterion assumption and it also shows lower sensitivity ($d'=0.2$ in the first and $d' = 0.3$ in the second task). This way, we favor confirming the [ITA]{} hypothesis. For the first indirect measure, we computed the $t$ value from the given estimates for the congruency effect ($M=24$ ms and $SD=13.5$). For the second indirect measure, the statistic ($t(11)<3$) is taken from Figure 4, where the covert activation reflects processing of the prime as opposed processing of the target in the overt activation. For the third indirect measure, we only considered the congruency effect on fMRI the results are provided in Figure 5.

  ----------------------------------- ---- --------------- ----------------- ------------------------------ ------------------------------------
                                                                                                            
  (l)[2-4]{} (l)[5-6]{}                  N   no. of trials Statistic           $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct, word vs. digit      7             112 $d' = 0.2$                         $0.20\pm0.11$ 
  Indirect, RT                          12             512 $t(11) = 6.16$                     $0.29\pm0.09$                        $0.09\pm0.14$
  Indirect, EEG (LRP)                   12             512 $t(11) = 3$                        $0.14\pm0.06$                       $-0.06\pm0.12$
  Indirect, fMRI                         9             128 $F(1,8) = 6.23$                    $0.17\pm0.10$                       $-0.03\pm0.14$
  ----------------------------------- ---- --------------- ----------------- ------------------------------ ------------------------------------

  : Reported results from Dehaene et al. (Nature, 1998)

### 

The study reports two experiments. In E1, multiple measures assessed the visibility of the masked stimulus and we chose the reported binary forced-choice task (no stimulus vs. masked stimulus) because it is the most relevant result. In this experiment, the [ITA]{} refers to the absence vs. presence of the masked stimuli. The fMRI results in E1 were NR. In E2, the [ITA]{} referred to the the congruency effect of repeated (congruent, either in same or in different case) vs. different words (incongruent).

SR: “Behaviorally, participants again denied seeing the primes and were unable to select them in a two-alternative forced-choice test \[...\]. However, case-independent repetition priming was observed in response times recorded during imaging \[...\]” (p. 755) and “As this phenomenon depends only on the identity of the masked prime, specific information about word identity must have been extracted and encoded unconsciously \[...\]” (p. 756).

  ------------------------------------- ---- --------------- ------------------ ------------------------------ ------------------------------------
                                                                                                               
  (l)[2-4]{} (l)[5-6]{}                    N   no. of trials Statistic            $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct (E1)                  27              36 52.9%-correct                       $0.15\pm0.09$ 
  Indirect, EEG, P1 (E1)                  12             300 $t(11) = 2.04$                      $0.10\pm0.06$                       $-0.04\pm0.10$
  Indirect, EEG, N1 (E1)                  12             300 $F(1,11) = 9.79$                    $0.16\pm0.07$                        $0.01\pm0.11$
  (l)[1-6]{} Direct (E2)                  10              64 53.6%-correct                       $0.18\pm0.11$ 
  Indirect, RT (E2)                       10             480 $F(1,9) = 36$                       $0.30\pm0.10$                        $0.12\pm0.15$
  Indirect, fMRI, same-case (E2)          10             240 $t(9) = 1.98$                       $0.34\pm0.11$                        $0.16\pm0.16$
  Indirect, fMRI, different-case (E2)     10             240 $t(9) = 2.68$                       $0.34\pm0.11$                        $0.16\pm0.16$
  ------------------------------------- ---- --------------- ------------------ ------------------------------ ------------------------------------

  : Reported results from Dehaene et al. (Nat. Neuro., 2001)

### 

The study reported four experiments. Prime and target stimuli were presented in different locations to the participants. In half of the trials the prime location was cued (pc) and in the other half it was the target location (tc). We excluded the target cued condition in E1 because it was NIE. In E3, multiple within factors were tested but since those do not change the reported $F$ value of the congruency effect we could nevertheless reanalyze it. E4 did not follow the standard reasoning.

  ---------------------------- ---- --------------- ------------------- ------------------------------ ------------------------------------
                                                                                                       
  (l)[2-4]{} (l)[5-6]{}           N   no. of trials Statistic             $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct, pc (E1)     40              80 $d' = 0.22$                          $0.22\pm0.05$ 
  Indirect, RT, pc (E1)          40              80 $F(1,39) = 33.94$                    $0.24\pm0.05$                        $0.02\pm0.07$
  (l)[1-6]{} Direct, pc (E2)     40              80 $d' = -0.06$                        $-0.06\pm0.05$ 
  Indirect, RT, pc (E2)          40              80 $F(1,39) = 8.5$                      $0.12\pm0.05$                        $0.18\pm0.07$
  (l)[1-6]{} Direct, tc (E2)     40              80 $d' = 0.07$                          $0.07\pm0.05$ 
  Indirect, RT, tc (E2)          40              80 $F(1,39) = 10.6$                     $0.14\pm0.05$                        $0.07\pm0.07$
  (l)[1-6]{} Direct (E3)         20             240 $d' = 0.05$                          $0.05\pm0.05$ 
  Indirect, RT (E3)              20             720 $F(1,19) = 31.37$                    $0.20\pm0.05$                        $0.15\pm0.07$
  ---------------------------- ---- --------------- ------------------- ------------------------------ ------------------------------------

  : Reported results from Finkbeiner & Palermo (Psych. Sci., 2009)

### 

The study presented trials in two conditions, one with a short ($40$ ms) and one with a long ($50$ ms) prime presentation duration. An [ITA]{} was concluded only for the short duration and with respect to the semantic content (not color).

SR: “In contrast, 16 of the 21 subjects were judged to be at chance with the 40-ms primes. Following Rouder et al. (2007), the RTs for the 17 subject-by-prime-duration combinations for which subliminality was confirmed were entered into a paired-samples $t$ test (two-tailed) to determine whether subliminal priming had occurred” (p. 1260).

  -------------------------- ---- --------------- --------------- ------------------------------ ------------------------------------
                                                                                                 
  (l)[2-4]{} (l)[5-6]{}         N   no. of trials Statistic         $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct, 40 ms     21             120 $d' = 0.098$                     $0.10\pm0.06$ 
  Indirect, RT, 40 ms          21              80 $t(20) = 2.5$                    $0.14\pm0.06$                        $0.04\pm0.09$
  -------------------------- ---- --------------- --------------- ------------------------------ ------------------------------------

  : Reported results from Finkbeiner (AP&P, 2011)

### 

The study reported two experiments. E1 reported the indirect task results and E2 reported the direct task results. In E1, indirect effects on RT, error rates and some EEG components were NR because the reported statistics combine masked and unmasked conditions (for unmasked conditions, they claimed no [ITA]{}) except for the N400 component in EEG. In E2, there were multiple direct tasks (see their Table 1). We chose the direct task on semantic judgment because the indirect task’s congruency effect was an effect from semantic relatedness too.

SR: “Average $d'$ measures in all tasks and context conditions did not deviate significantly from zero demonstrating that masked words were not identified” (p. 36).

  ---------------------------------- ---- --------------- ------------------ ------------------------------ ------------------------------------
                                                                                                            
  (l)[2-4]{} (l)[5-6]{}                 N   no. of trials Statistic            $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct, semantic (E2)     24              80 $d' = 0.14$                         $0.14\pm0.06$ 
  Indirect, EEG, N400 (E1)             24             320 $F(1,23) = 5.48$                    $0.09\pm0.04$                       $-0.05\pm0.08$
  ---------------------------------- ---- --------------- ------------------ ------------------------------ ------------------------------------

  : Reported results from Kiefer (Cog. Brain Res., 2002)

### 

The study reported four experiments. In E1, there were multiple direct task measures from which we chose the one that fit our model assumptions of a neutral criterion (the identification rate is not comparable by our method). Also in E1, we chose not to consider sub-analyses of the indirect effects because they are essentially repetitions of the same comparison. In E2, we did not consider the non-target set condition and in E3 we did not consider the error rate analysis as they were NIE. In E1-E3, trials with neutral primes were not considered for calculating the priming effect.

SR: “The identification rate for the prime numbers was 2.2% (the chance level is 6.25% as each prime is presented four times in the 64 test trials). Thus, the primes were indeed unidentifiable, as is usually found under the experimental conditions that we adopted (Damian, 2001; Dehaene et al., 1998; Koechlin et al., 1999; Naccache & Dehaene, 2001)” (p. 230).

  --------------------------------------- ---- --------------- ------------------- ------------------------------ ------------------------------------
                                                                                                                  
  (l)[2-4]{} (l)[5-6]{}                      N   no. of trials Statistic             $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct (E1)                    12              64 $d' = 0.29$                          $0.29\pm0.10$ 
  Indirect, RT (E1)                         12            1152 $F(1,11) = 25.17$                    $0.22\pm0.07$                       $-0.07\pm0.12$
  (l)[1-6]{} Direct (E2)                    12              64 $d' = 0.33$                          $0.33\pm0.10$ 
  Indirect, RT, target set (E2)             12             288 $F(1,11) = 15.24$                    $0.20\pm0.07$                       $-0.13\pm0.12$
  Indirect, error rate, target set (E2)     12             288 $F(1,11) = 6.35$                     $0.13\pm0.06$                       $-0.20\pm0.12$
  (l)[1-6]{} Direct (E3)                    12              64 $d' = -0.11$                        $-0.11\pm0.10$ 
  Indirect, RT, target set (E3)             12             144 $F(1,11) = 21.67$                    $0.28\pm0.09$                        $0.39\pm0.14$
  Indirect, RT, non-target set (E3)         12             144 $F(1,11) = 6.58$                     $0.15\pm0.08$                        $0.26\pm0.13$
  (l)[1-6]{} Direct (E4)                    24              64 $d' = 0.22$                          $0.22\pm0.07$ 
  Indirect, RT (E4)                         24            1152 $F(1,23) = 43.2$                     $0.21\pm0.05$                       $-0.01\pm0.08$
  Indirect, error rate (E4)                 24            1152 $F(1,23) = 9.17$                     $0.10\pm0.04$                       $-0.12\pm0.08$
  --------------------------------------- ---- --------------- ------------------- ------------------------------ ------------------------------------

  : Reported results from Kunde et al. (Cogn., 2003)

### 

The study reports five experiments. Only Experiments 3 and 5 are considered to be evidence for unconscious priming. Experiment 3 suffers severely from RTM and therefore not reanalyzed.

SR: “We might assume that performance at chance level indexes absence of all conscious information. This assumption was made in a number of studies (e.g., Dehaene et al., 1998; Klotz & Neumann, 1999; Neumann & Klotz, 1994; Vorberg et al., in press). In the present study, evidence for priming without awareness comes from Experiment 3 and Experiment 5, in which participants showed substantial non-motor priming effects although they could not discriminate primes better than chance” (p. 184)

  ------------------------ ---- --------------- ------------------ ------------------------------ ------------------------------------
                                                                                                  
  (l)[2-4]{} (l)[5-6]{}       N   no. of trials Statistic            $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct (E5)     11             320 $d' = 0.28$                         $0.28\pm0.06$ 
  Indirect, RT (E5)          11             320 $F(1,10) = 18.5$                    $0.22\pm0.08$                       $-0.06\pm0.10$
  ------------------------ ---- --------------- ------------------ ------------------------------ ------------------------------------

  : Reported results from Mattler (P & P., 2003)

### 

The study reports two experiments. For the direct tasks in both experiments, the authors additionally conducted the Greenwald method which has been criticized before [@Klauer_etal_98_comment; @Merikle_Reingold_98]. Therefore, we only considered typical results as in all other studies. We considered only the main congruency effects on RT and no further subanalyses because the reported direct task would not have been comparable.

SR: “In this task, subjects performed at chance level, while priming effects were replicated. This study provides strong evidence for the unconscious nature of our semantic priming effects” (p. 227).

  ------------------------ ---- --------------- ------------------- ------------------------------ ------------------------------------
                                                                                                   
  (l)[2-4]{} (l)[5-6]{}       N   no. of trials Statistic             $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct (E1)     18              32 $d' = 0.6$                           $0.60\pm0.11$ 
  Indirect, RT (E1)          18             384 $F(1,17) = 21.99$                    $0.19\pm0.06$                       $-0.41\pm0.12$
  (l)[1-6]{} Direct (E2)     18              64 $d' = 0.01$                          $0.01\pm0.08$ 
  Indirect, RT (E2)          18             384 $F(1,17) = 21.62$                    $0.19\pm0.06$                        $0.18\pm0.10$
  ------------------------ ---- --------------- ------------------- ------------------------------ ------------------------------------

  : Reported results from Naccache & Dehaene (Cogn., 2001)

### 

The study reported three experiments. We did not consider the subanalyses for cued trials as the standard reasoning only related to the congruency effects. Note that we only counted the number of “critical” trials which were used in their analysis.

  ---------------------------------- ---- --------------- ------------------ ------------------------------ ------------------------------------
                                                                                                            
  (l)[2-4]{} (l)[5-6]{}                 N   no. of trials Statistic            $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct (E1)               12             240 $d' = -0.04$                       $-0.04\pm0.06$ 
  Indirect, RT (E1)                    12             240 $F(1,11) = 7.88$                    $0.15\pm0.07$                        $0.19\pm0.09$
  (l)[1-6]{} Direct (E2)               12             240 $d' = -0.07$                       $-0.07\pm0.06$ 
  Indirect, RT (E1)                    12             240 $F(1,11) = 7.32$                    $0.14\pm0.07$                        $0.21\pm0.09$
  (l)[1-6]{} Direct (E3)               12             240 $d' = 0.09$                         $0.09\pm0.06$ 
  Indirect, RT, early, valid (E3)      12             240 $F(1,11) = 9.23$                    $0.16\pm0.07$                        $0.07\pm0.09$
  Indirect, RT, late, valid (E3)       12             240 $F(1,11) = 3.97$                    $0.11\pm0.06$                        $0.02\pm0.09$
  Indirect, RT, late, invalid (E3)     12             240 $F(1,11) = 5.34$                    $0.12\pm0.07$                        $0.03\pm0.09$
  ---------------------------------- ---- --------------- ------------------ ------------------------------ ------------------------------------

  : Reported results from Naccache et al. (Psych. Sci., 2002)

### 

The study deviated from the standard priming paradigm by just showing masked stimuli (in this case, coins) and no target stimuli. Presentation duration was varied in three conditions. For the separate conditions, participants were measured in one direct task and with three indirect measures. The appendix provided the required information for our reanalysis. We digitized their Figure S2 to derive at the $t$ values for the two indirect measures grip force and pallidal activation. The third indirect measure, skin conductance, was NIE. Even though these results were only reported in the appendix, the study bases their interpretation on these results Note that $N=24$ relates to 24 participant $\times$ stimulus duration conditions in which the direct task was non-significant at an individual level. RTM. SR: “Based on the percentage of correct responses, the analysis could then be restricted to all situations where subjects guess at chance level about stimulus identity (fig. S2). Even in these situations, pallidal activation and hand-grip force were significantly higher for pounds as compared to pennies \[...\] These results indicate that motivational processes involved in boosting behavior are qualitatively similar, despite whether subjects are conscious or not of the reward at stake.” (p. 906).

  ---------------------------------- ---- --------------- ---------------- ------------------------------ ------------------------------------
                                                                                                          
  (l)[2-4]{} (l)[5-6]{}                 N   no. of trials Statistic          $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct, Semantic (E2)     24              60 $d' = 0.19$                       $0.19\pm0.07$ 
  Indirect, grip force                 24              90 $t(23) = 2.92$                    $0.15\pm0.06$                       $-0.04\pm0.09$
  Indirect, pallidal activation        24              90 $t(23) = 3.41$                    $0.17\pm0.06$                       $-0.02\pm0.09$
  ---------------------------------- ---- --------------- ---------------- ------------------------------ ------------------------------------

  : Reported results from Pessiglione et al. (Science, 2007)

### 

The study reported two experiments. Both, E1 and E2, had different mask conditions (A vs. B). Only E1 provided the indirect task results such that we could reanalyze both conditions separately. For E2 we had to apply our reanalysis to both conditions aggregated. Therefore, we averaged over the given $d'$ values from both conditions. We did not consider the subanalyses on the difference and interaction between the two masks but only to the congruency effects as done in the standard reasoning. RTM.

  -------------------------------- ---- --------------- --------------- ------------------------------ ------------------------------------
                                                                                                       
  (l)[2-4]{} (l)[5-6]{}               N   no. of trials Statistic         $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct, mask A (E1)     12              40 $d' = 0.14$                      $0.14\pm0.12$ 
  Indirect, RT, mask A (E1)          12             200 $t(11) = 5.5$                    $0.30\pm0.10$                        $0.16\pm0.15$
  (l)[1-6]{} Direct, mask B (E1)     12              40 $d' = 0.11$                      $0.11\pm0.12$ 
  Indirect, RT, mask B (E1)          12             200 $t(11) = 4.5$                    $0.25\pm0.09$                        $0.14\pm0.15$
  (l)[1-6]{} Direct, mask B (E2)     12              80 50.5%-correct                    $0.03\pm0.09$ 
  Indirect, RT (E2)                  12             400 $t(11) = 7.4$                    $0.36\pm0.10$                        $0.33\pm0.14$
  Indirect, error rate (E2)          12             400 $t(11) = 4$                      $0.19\pm0.07$                        $0.17\pm0.12$
  -------------------------------- ---- --------------- --------------- ------------------------------ ------------------------------------

  : Reported results from Sumner (Exp. Psych., 2008)

### 

The study reported one experiment with one direct task and multiple indirect measures. However, we only considered the indirect effect on RTs as the fMRI analyses were NR. RTM.

SR: “\[...\] a, Participants were unable to discriminate between trials with a strongly masked square or diamond, as revealed by chance-level performance in a two-choice discrimination task administered after the main experiment. b, Although strongly masked no-go signals could not be perceived consciously, they still triggered inhibitory control processes, as revealed by significantly longer response times on these trials than on strongly masked go trials.” (in Figure 2, p. 4145).

  ----------------------- ---- --------------- ---------------- ------------------------------ ------------------------------------
                                                                                               
  (l)[2-4]{} (l)[5-6]{}      N   no. of trials Statistic          $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct         20              48 $d' = 0.118$                      $0.12\pm0.09$ 
  Indirect, RT              20             240 $t(19) = 6.24$                    $0.27\pm0.06$                        $0.15\pm0.11$
  ----------------------- ---- --------------- ---------------- ------------------------------ ------------------------------------

  : Reported results from van Gaal et al. (JoNeuro, 2010)

### 

The study reported two experiments. In E1, there were two outline conditions, line vs. rectangle. The line condition yielded a negative congruency effect which we treated similar to a standard (positive) priming effect. The rectangle condition was NIE. In E2, the rectangle condition with prime duration of $50$ ms produced a large $d'$ so that no [ITA]{} was claimed. Hence, we only considered the rectangle condition only for $33$ ms. For the line condition, $33$ ms and $50$ ms trials were analyzed together since there was no interaction effect.

SR: “The results from the FC task indicated that similar prime visibility, equivalent to chance level, was obtained in the two preposed object type conditions. This finding confirmed that primes were processed subliminally in the primary task” (p. 425).

  --------------------------------------------- ---- --------------- ------------------- ------------------------------ ------------------------------------
                                                                                                                        
  (l)[2-4]{} (l)[5-6]{}                            N   no. of trials Statistic             $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct (E1)                          15              64 $d' = 0.02$                          $0.02\pm0.09$ 
  Indirect, RT, line (E1)                         15             208 $F(1,14) = 6.86$                     $0.13\pm0.06$                        $0.11\pm0.11$
  (l)[1-6]{} Direct, 33 ms, rect. (E2)            15              32 $d' = 0.15$                          $0.15\pm0.12$ 
  Indirect, RT, 33 ms, rect. (E2)                 15             208 $F(1,14) = 8.15$                     $0.14\pm0.06$                       $-0.01\pm0.14$
  (l)[1-6]{} Direct, 33 ms + 50 ms, line (E2)     15              64 $d' = -0.103$                       $-0.10\pm0.09$ 
  Indirect, RT, 33 ms + 50 ms, line (E2)          15             416 $F(1,14) = 11.47$                    $0.15\pm0.06$                        $0.25\pm0.11$
  --------------------------------------------- ---- --------------- ------------------- ------------------------------ ------------------------------------

  : Reported results from Wang et al. (Exp. Psych., 2017)

### 

The study reported one experiment with masked and unmasked conditions. We only considered the masked condition for which an [ITA]{} was claimed but not the unmasked condition. In the direct task, we had to compute average $d'$ from the openly accessible material. In the indirect task, EEG components were measured. For EEG preprocessing, some trials had to be rejected leading to an average of 131 trials.

SR: “Analysis of the sensitivity measure $d'$ indicated that faces were not consciously identified in the masked condition. A clear N2 posterior-contralateral (N2pc) component (a neural marker of attention shifts) was found in both the masked and unmasked conditions, revealing that one’s own face automatically captures attention when processed unconsciously” (in the abstract, p. 471).

  ----------------------------- ---- --------------- ---------------- ------------------------------ ------------------------------------
                                                                                                     
  (l)[2-4]{} (l)[5-6]{}            N   no. of trials Statistic          $d'_\text{estimated} \pm SE$   $d'_\text{diff}\pm SE_\text{diff}$
  (l)[1-6]{} Direct, masked       18             160 $d' = 0.211$                      $0.21\pm0.06$ 
  Indirect, EEG, N2pc, masked     18             131 $t(17) = 2.34$                    $0.12\pm0.06$                       $-0.09\pm0.08$
  ----------------------------- ---- --------------- ---------------- ------------------------------ ------------------------------------

  : Reported results from Wójcik et al. (Psych. Sci., 2019)

**[Glossary]{}** {#app:symbols}
================

  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Variable                Description
  ----------------------- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
  $c_j$                   Condition effect, for example the congruent condition ($j=1$) produces faster RTs so that $c_1$.

  $d' $                   Observed, average sensitivity index, measures the true sensitivity $d'_\text{true}$.

  $d'_i$                  Observed, individual sensitivity indices, measures the true, individual sensitivities $d'_{\text{true},i}$.

  $d'_\text{true}$        True sensitivity, $d'_\text{true} = \frac{\Delta}{\sigma_\epsilon}$.

  $d'_\text{estimated}$   Estimated sensitivity from the reported results in the direct ($d'_\text{estimated,direct}$) or indirect task ($d'_\text{estimated,indirect}$).

  $d'_{\text{true},i}$    Individual sensitivity, $d'_{\text{true},i} = \frac{\Delta_i}{\sigma_\epsilon}$.

  $\Delta$                The true difference between conditions, $\Delta =
                                  c_2 - c_1$.

  $\hat{\Delta}$          The observed, mean difference between conditions.

  $\Delta_i$              Individual effects, $\Delta_i = c_2 + (p\times c)_{i2} - (c_1 + (p\times c)_{i1})$, for example the expected congruency effect between conditions of participant $i$.

  $\hat{\Delta}_i$        The observed difference between conditions of participant $i$.

  $\epsilon_{ijk}$        trial-by-trial error, noise due to measurement error or random neuronal fluctuations.

  $f_{opt}(x)$            Optimal classifier taking indirect measures $x$ (e.g., RTs) and predicting the condition (congruent/incongruent).

  $f_{t}(x)$              Threshold classifier taking predicting one condition for indirect measures $x\leq t$ (e.g., RTs) and the other for $x>t$.

  $h$                     Approximation used to translate between sensitivities and accuracies.

  $i$                     Index for participant $i \in \{1,2,...,N\}$.

  $j$                     Index for condition $j \in \{1,2\}$, for example indicator for congruent ($j=1$) and incongruent ($j=2$) conditions.

  $k$                     Index for trial $k \in \{1,2,...,M\}$. Since there are two conditions, the number of observed trials per participant is $2M$.

  $\kappa_{N,M,q^2}$      Constant relating $t$ values to the estimated sensitivity in the indirect task, $d'_{\text{true}} = \kappa_{N,M,q^2} \cdot t$. It depends on $N$, $M$ and $q$.

  $M$                     Number of trials per participant $\times$ condition. The total number of trials per participant is $2M$.

  $\mu$                   Grand mean, for example the overall expected value of RTs.

  $N$                     Number of participants.

  $\Omega(x)$             Marginal, cumulative density distribution (CDF) over indirect measures $x$.
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  Variable                          Description
  --------------------------------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  $p_i$                             Participant effect, for example participants with a faster RTs than average have a negative $p_i$ while slower participants have a positive $p_i$.
  $(p\times c)_{ij}$                Interaction effect, for example some participants have different reaction time effects.
  $\pi$                             True accuracy.
  $\hat{\pi}$                       Observed, mean accuracy.
  $\pi_i$                           True accuracy of participant $i$. It is can be translated into a sensitivity by $d'_{\text{true},i} = 2\Phi^{-1}(\pi_i)$ where $\Phi$ is the cumulative normal distribution.
  $\hat{\pi}_i$                     Observed, individual accuracy.
  $q^2$                             Ratio between effect variance and trial-by-trial error variance, $q^2~=~\frac{\sigma^2_\text{effect}}{\sigma^2_\epsilon}$. This is the variance of true sensitivities across individuals, $q^2 = \text{Var}[d'_{\text{true},i}]$. A reasonable value in our setting is $q^2 = 0.0225$ implying $\text{SD}[d'_{\text{true},i}] = 0.15$.
  $SE$                              Estimated standard error of the estimated sensitivity.
  $\sigma^2_{{\Delta}_i}$           Variance of true individual effects, for example, to which degree participants vary in their congruency effect.
  $\sigma^2_{\hat{\Delta}_i}$       True variance of observed individual effects, for example, variance of the observable congruency effects.
  $\hat\sigma^2_{\hat{\Delta}_i}$   Estimated variance of observed individual effects. This is what scientists get when computing the variance on the observable congruency effects across participants.
  $\sigma^2_{p\times c}$            Variance of the interaction effect, $(p \times c)_{ij}$.
  $\sigma^2_\text{effect}$          Variance of the effects $\Delta_i$, $\sigma^2_\text{effect} = 4\sigma^2_{p\times c}$.
  $\sigma^2_\epsilon$               Variance of the trial-by-trial error, $\epsilon_{ijk}$.
  $t$                               $t$ value, in our context it comes from paired-$t$-tests between the two conditions of the indirect task.
  $Y_{ijk}$                         Response of participant $i$ in condition $j$ trial $k$ from the direct ($Y_{ijk}^{\text{dir}}$) or indirect task ($Y_{ijk}^{\text{indir}}$). The standard repeated measures ANOVA model is $Y_{ijk}~=~\mu~+~p_i~+~c_j~+~(p\times c)_{ij}~+~\epsilon_{ijk}$.
