  Variable selection is essential for improving inference and interpretation in
multivariate linear regression. Although a number of alternative regressor
selection criteria have been suggested, the most prominent and widely used are
the Akaike information criterion (AIC), Bayesian information criterion (BIC),
Mallow's $C_p$, and their modifications. However, for high-dimensional data,
experience has shown that the performance of these classical criteria is not
always satisfactory. In the present article, we begin by presenting the
necessary and sufficient conditions (NSC) for the strong consistency of the
high-dimensional AIC, BIC, and $C_p$, based on which we can identify some
reasons for their poor performance. Specifically, we show that under certain
mild high-dimensional conditions, if the BIC is strongly consistent, then the
AIC is strongly consistent, but not vice versa. This result contradicts the
classical understanding. In addition, we consider some NSC for the strong
consistency of the high-dimensional kick-one-out (KOO) methods introduced by
Zhao et al. (1986) and Nishii et al. (1988). Furthermore, we propose two
general methods based on the KOO methods and prove their strong consistency.
The proposed general methods remove the penalties while simultaneously reducing
the conditions for the dimensions and sizes of the regressors. A simulation
study supports our consistency conclusions and shows that the convergence rates
of the two proposed general KOO methods are much faster than those of the
original methods.
