  Sparse linear (or generalized linear) models combine a standard likelihood
function with a sparse prior on the unknown coefficients. These priors can
conveniently be expressed as a maximization over zero-mean Gaussians with
different variance hyperparameters. Standard MAP estimation (Type I) involves
maximizing over both the hyperparameters and coefficients, while an empirical
Bayesian alternative (Type II) first marginalizes the coefficients and then
maximizes over the hyperparameters, leading to a tractable posterior
approximation. The underlying cost functions can be related via a dual-space
framework from Wipf et al. (2011), which allows both the Type I or Type II
objectives to be expressed in either coefficient or hyperparmeter space. This
perspective is useful because some analyses or extensions are more conducive to
development in one space or the other. Herein we consider the estimation of a
trade-off parameter balancing sparsity and data fit. As this parameter is
effectively a variance, natural estimators exist by assessing the problem in
hyperparameter (variance) space, transitioning natural ideas from Type II to
solve what is much less intuitive for Type I. In contrast, for analyses of
update rules and sparsity properties of local and global solutions, as well as
extensions to more general likelihood models, we can leverage coefficient-space
techniques developed for Type I and apply them to Type II. For example, this
allows us to prove that Type II-inspired techniques can be successful
recovering sparse coefficients when unfavorable restricted isometry properties
(RIP) lead to failure of popular L1 reconstructions. It also facilitates the
analysis of Type II when non-Gaussian likelihood models lead to intractable
integrations.
