  Many testing problems are readily amenable to randomised tests such as those
employing data splitting. However despite their usefulness in principle,
randomised tests have obvious drawbacks. Firstly, two analyses of the same
dataset may lead to different results. Secondly, the test typically loses power
because it does not fully utilise the entire sample. As a remedy to these
drawbacks, we study how to combine the test statistics or p-values resulting
from multiple random realisations such as through random data splits. We
develop rank-transformed subsampling as a general method for delivering large
sample inference about the combined statistic or p-value under mild
assumptions. We apply our methodology to a wide range of problems, including
testing unimodality in high-dimensional data, testing goodness-of-fit of
parametric quantile regression models, testing no direct effect in a
sequentially randomised trial and calibrating cross-fit double machine learning
confidence intervals. In contrast to existing p-value aggregation schemes that
can be highly conservative, our method enjoys type-I error control that
asymptotically approaches the nominal level. Moreover, compared to using the
ordinary subsampling, we show that our rank transform can remove the
first-order bias in approximating the null under alternatives and greatly
improve power.
