  We consider the problem of testing whether a single coefficient is equal to
zero in fixed-design linear models under a moderately high-dimensional regime,
where the dimension of covariates $p$ is allowed to be in the same order of
magnitude as sample size $n$. In this regime, to achieve finite-population
validity, existing methods usually require strong distributional assumptions on
the noise vector (such as Gaussian or rotationally invariant), which limits
their applications in practice. In this paper, we propose a new method, called
residual permutation test (RPT), which is constructed by projecting the
regression residuals onto the space orthogonal to the union of the column
spaces of the original and permuted design matrices. RPT can be proved to
achieve finite-population size validity under fixed design with just
exchangeable noises, whenever $p < n / 2$. Moreover, RPT is shown to be
asymptotically powerful for heavy tailed noises with bounded $(1+t)$-th order
moment when the true coefficient is at least of order $n^{-t/(1+t)}$ for $t \in
[0,1]$. We further proved that this signal size requirement is essentially
rate-optimal in the minimax sense. Numerical studies confirm that RPT performs
well in a wide range of simulation settings with normal and heavy-tailed noise
distributions.
