  Fisher's randomization test (FRT) delivers exact $p$-values under the strong
null hypothesis of no treatment effect on any units whatsoever and allows for
flexible covariate adjustment to improve the power. Of interest is whether the
procedure could also be valid for testing the weak null hypothesis of zero
average treatment effect. Towards this end, we evaluate two general strategies
for FRT with covariate-adjusted test statistics: that based on the residuals
from an outcome model with only the covariates, and that based on the output
from an outcome model with both the treatment and the covariates. Based on
theory and simulation, we recommend using the ordinary least squares (OLS) fit
of the observed outcome on the treatment, centered covariates, and their
interactions for covariate adjustment, and conducting FRT with the robust
$t$-value of the treatment as the test statistic. The resulting FRT is
finite-sample exact for the strong null hypothesis, asymptotically valid for
the weak null hypothesis, and more powerful than the unadjusted analog under
alternatives, all irrespective of whether the linear model is correctly
specified or not. We develop the theory for complete randomization, cluster
randomization, stratified randomization, and rerandomization, respectively, and
give a recommendation for the test procedure and test statistic under each
design. We first focus on the finite-population perspective and then extend the
result to the super-population perspective, highlighting the difference in
standard errors. Motivated by the similarity in procedure, we also evaluate the
design-based properties of five existing permutation tests originally for
linear models and show the superiority of the proposed FRT for testing the
treatment effects.
