  Regression models are used for inference and prediction in a wide range of
applications providing a powerful scientific tool for researchers and analysts
from different fields. In many research fields the amount of available data as
well as the number of potential explanatory variables is rapidly increasing.
Variable selection and model averaging have become extremely important tools
for improving inference and prediction. However, often linear models are not
sufficient and the complex relationship between input variables and a response
is better described by introducing non-linearities and complex functional
interactions. Deep learning models have been extremely successful in terms of
prediction although they are often difficult to specify and potentially suffer
from overfitting. The aim of this paper is to bring the ideas of deep learning
into a statistical framework which yields more parsimonious models and allows
to quantify model uncertainty. To this end we introduce the class of deep
Bayesian regression models (DBRM) consisting of a generalized linear model
combined with a comprehensive non-linear feature space, where non-linear
features are generated just like in deep learning but combined with variable
selection in order to include only important features. DBRM can easily be
extended to include latent Gaussian variables to model complex correlation
structures between observations, which seems to be not easily possible with
existing deep learning approaches. Two different algorithms based on MCMC are
introduced to fit DBRM and to perform Bayesian inference. The predictive
performance of these algorithms is compared with a large number of state of the
art algorithms. Furthermore we illustrate how DBRM can be used for model
inference in various applications.
