  Scientists often adjust their significance threshold (alpha level) during
null hypothesis significance testing in order to take into account multiple
testing and multiple comparisons. This alpha adjustment has become particularly
relevant in the context of the replication crisis in science. The present
article considers the conditions in which this alpha adjustment is appropriate
and the conditions in which it is inappropriate. A distinction is drawn between
three types of multiple testing: disjunction testing, conjunction testing, and
individual testing. It is argued that alpha adjustment is only appropriate in
the case of disjunction testing, in which at least one test result must be
significant in order to reject the associated joint null hypothesis. Alpha
adjustment is inappropriate in the case of conjunction testing, in which all
relevant results must be significant in order to reject the joint null
hypothesis. Alpha adjustment is also inappropriate in the case of individual
testing, in which each individual result must be significant in order to reject
each associated individual null hypothesis. The conditions under which each of
these three types of multiple testing is warranted are examined. It is
concluded that researchers should not automatically (mindlessly) assume that
alpha adjustment is necessary during multiple testing. Illustrations are
provided in relation to joint studywise hypotheses and joint multiway ANOVAwise
hypotheses.
