  Consider testing multiple hypotheses using tests that can only be evaluated
by simulation, such as permutation tests or bootstrap tests. This article
introduces MMCTest, a sequential algorithm which gives, with arbitrarily high
probability, the same classification as a specific multiple testing procedure
applied to ideal p-values. The method can be used with a class of multiple
testing procedures which includes the Benjamini & Hochberg False Discovery Rate
(FDR) procedure and the Bonferroni correction controlling the Familywise Error
Rate. One of the key features of the algorithm is that it stops sampling for
all the hypotheses which can already be decided as being rejected or
non-rejected. MMCTest can be interrupted at any stage and then returns three
sets of hypotheses: the rejected, the non-rejected and the undecided
hypotheses. A simulation study motivated by actual biological data shows that
MMCTest is usable in practice and that, despite the additional guarantee, it
can be computationally more efficient than other methods.
