  Modern social and biomedical scientific publications require the reporting of
covariate balance tables with not only covariate means by treatment group but
also the associated $p$-values from significance tests of their differences.
The practical need to avoid small $p$-values renders balance check and
rerandomization by hypothesis testing standards an attractive tool for
improving covariate balance in randomized experiments. Despite the
intuitiveness of such practice and its arguably already widespread use in
reality, the existing literature knows little about its implications on
subsequent inference, subjecting many effectively rerandomized experiments to
possibly inefficient analyses. To fill this gap, we examine a variety of
potentially useful schemes for rerandomization based on $p$-values (ReP) from
covariate balance tests, and demonstrate their impact on subsequent inference.
Specifically, we focus on three estimators of the average treatment effect from
the unadjusted, additive, and fully interacted linear regressions of the
outcome on treatment, respectively, and derive their respective asymptotic
sampling properties under ReP. The main findings are twofold. First, the
estimator from the fully interacted regression is asymptotically the most
efficient under all ReP schemes examined, and permits convenient
regression-assisted inference identical to that under complete randomization.
Second, ReP improves not only covariate balance but also the efficiency of the
estimators from the unadjusted and additive regressions asymptotically. The
standard regression analysis, in consequence, is still valid but can be overly
conservative.
