  Penalization schemes like Lasso or ridge regression are routinely used to
regress a response of interest on a high-dimensional set of potential
predictors. Despite being decisive, the question of the relative strength of
penalization is often glossed over and only implicitly determined by the scale
of individual predictors. At the same time, additional information on the
predictors is available in many applications but left unused. Here, we propose
to make use of such external covariates to adapt the penalization in a
data-driven manner. We present a method that differentially penalizes feature
groups defined by the covariates and adapts the relative strength of
penalization to the information content of each group. Using techniques from
the Bayesian tool-set our procedure combines shrinkage with feature selection
and provides a scalable optimization scheme. We demonstrate in simulations that
the method accurately recovers the true effect sizes and sparsity patterns per
feature group. Furthermore, it leads to an improved prediction performance in
situations where the groups have strong differences in dynamic range. In
applications to data from high-throughput biology, the method enables
re-weighting the importance of feature groups from different assays. Overall,
using available covariates extends the range of applications of penalized
regression, improves model interpretability and can improve prediction
performance. We provide an open-source implementation of the method in the R
package graper.
