  We consider penalized regression models under a unified framework where the
particular method is determined by the form of the penalty term. We propose a
fully Bayesian approach that incorporates both sparse and dense settings and
show how to use a type of model averaging approach to eliminate the nuisance
penalty parameters and perform inference through the marginal posterior
distribution of the regression coefficients. We establish tail robustness of
the resulting estimator as well as conditional and marginal posterior
consistency. We develop an efficient component-wise Markov chain Monte Carlo
algorithm for sampling. Numerical results show that the method tends to select
the optimal penalty and performs well in both variable selection and prediction
and is comparable to, and often better than alternative methods. Both simulated
and real data examples are provided.
