  Model selection is difficult to analyse yet theoretically and empirically
important, especially for high-dimensional data analysis. Recently the least
absolute shrinkage and selection operator (Lasso) has been applied in the
statistical and econometric literature. Consis- tency of Lasso has been
established under various conditions, some of which are difficult to verify in
practice. In this paper, we study model selection from the perspective of
generalization ability, under the framework of structural risk minimization
(SRM) and Vapnik-Chervonenkis (VC) theory. The approach emphasizes the balance
between the in-sample and out-of-sample fit, which can be achieved by using
cross-validation to select a penalty on model complexity. We show that an exact
relationship exists between the generalization ability of a model and model
selection consistency. By implementing SRM and the VC inequality, we show that
Lasso is L2-consistent for model selection under assumptions similar to those
imposed on OLS. Furthermore, we derive a probabilistic bound for the distance
between the penalized extremum estimator and the extremum estimator without
penalty, which is dominated by overfitting. We also propose a new measurement
of overfitting, GR2, based on generalization ability, that converges to zero if
model selection is consistent. Using simulations, we demonstrate that the
proposed CV-Lasso algorithm performs well in terms of model selection and
overfitting control.
