  In several large-scale replication projects, statistically non-significant
results in both the original and the replication study have been interpreted as
a "replication success". Here we discuss the logical problems with this
approach: Non-significance in both studies does not ensure that the studies
provide evidence for the absence of an effect and "replication success" can
virtually always be achieved if the sample sizes are small enough. In addition,
the relevant error rates are not controlled. We show how methods, such as
equivalence testing and Bayes factors, can be used to adequately quantify the
evidence for the absence of an effect and how they can be applied in the
replication setting. Using data from the Reproducibility Project: Cancer
Biology, the Experimental Philosophy Replicability Project, and the
Reproducibility Project: Psychology we illustrate that many original and
replication studies with "null results" are in fact inconclusive. We conclude
that it is important to also replicate studies with statistically
non-significant results, but that they should be designed, analyzed, and
interpreted appropriately.
