  We consider a statistical test whose p-value can only be approximated using
Monte Carlo simulations. We are interested in deciding whether the p-value for
an observed data set lies above or below a given threshold such as 5%. We want
to ensure that the resampling risk, the probability of the (Monte Carlo)
decision being different from the true decision, is uniformly bounded. This
article introduces a simple open-ended method with this property, the
confidence sequence method (CSM). We compare our approach to another algorithm,
SIMCTEST, which also guarantees an (asymptotic) uniform bound on the resampling
risk, as well as to other Monte Carlo procedures without a uniform bound. CSM
is free of tuning parameters and conservative. It has the same theoretical
guarantee as SIMCTEST and, in many settings, similar stopping boundaries. As it
is much simpler than other methods, CSM is a useful method for practical
applications.
