  We propose an approach for fitting linear regression models that splits the
set of covariates into groups. The optimal split of the variables into groups
and the regularized estimation of the regression coefficients are performed by
minimizing an objective function that encourages sparsity within each group and
diversity among them. The estimated coefficients are then pooled together to
form the final fit. Our procedure works on top of a given penalized linear
regression estimator (e.g., Lasso, elastic net) by fitting it to possibly
overlapping groups of features, encouraging diversity among these groups to
reduce the correlation of the corresponding predictions. For the case of two
groups, elastic net penalty and orthogonal predictors, we give a closed form
solution for the regression coefficients in each group. We establish the
consistency of our method with the number of predictors possibly increasing
with the sample size. An extensive simulation study and real-data applications
show that in general the proposed method improves the prediction accuracy of
the base estimator used in the procedure. Possible extensions to GLMs and other
models are discussed. The supplemental material for this article, available
online, contains the proofs of our theoretical results and the full results of
our simulation study.
