  Common high-dimensional methods for prediction rely on having either a sparse
signal model, a model in which most parameters are zero and there are a small
number of non-zero parameters that are large in magnitude, or a dense signal
model, a model with no large parameters and very many small non-zero
parameters. We consider a generalization of these two basic models, termed here
a "sparse+dense" model, in which the signal is given by the sum of a sparse
signal and a dense signal. Such a structure poses problems for traditional
sparse estimators, such as the lasso, and for traditional dense estimation
methods, such as ridge estimation. We propose a new penalization-based method,
called lava, which is computationally efficient. With suitable choices of
penalty parameters, the proposed method strictly dominates both lasso and
ridge. We derive analytic expressions for the finite-sample risk function of
the lava estimator in the Gaussian sequence model. We also provide an deviation
bound for the prediction risk in the Gaussian regression model with fixed
design. In both cases, we provide Stein's unbiased estimator for lava's
prediction risk. A simulation example compares the performance of lava to
lasso, ridge, and elastic net in a regression example using feasible,
data-dependent penalty parameters and illustrates lava's improved performance
relative to these benchmarks.
