  Regression models are used in a wide range of applications providing a
powerful scientific tool for researchers from different fields. Linear, or
simple parametric, models are often not sufficient to describe complex
relationships between input variables and a response. Such relationships can be
better described through flexible approaches such as neural networks, but this
results in less interpretable models and potential overfitting. Alternatively,
specific parametric nonlinear functions can be used, but the specification of
such functions is in general complicated. In this paper, we introduce a
flexible approach for the construction and selection of highly flexible
nonlinear parametric regression models. Nonlinear features are generated
hierarchically, similarly to deep learning, but have additional flexibility on
the possible types of features to be considered. This flexibility, combined
with variable selection, allows us to find a small set of important features
and thereby more interpretable models. Within the space of possible functions,
a Bayesian approach, introducing priors for functions based on their
complexity, is considered. A genetically modified mode jumping Markov chain
Monte Carlo algorithm is adopted to perform Bayesian inference and estimate
posterior probabilities for model averaging. In various applications, we
illustrate how our approach is used to obtain meaningful nonlinear models.
Additionally, we compare its predictive performance with several machine
learning algorithms.
