  Frequentist and Bayesian methods differ in many aspects, but share some basic
optimal properties. In real-life classification and regression problems,
situations exist in which a model based on one of the methods is preferable
based on some subjective criterion. Nonparametric classification and regression
techniques, such as decision trees and neural networks, have frequentist
(classification and regression trees (CART) and artificial neural networks) as
well as Bayesian (Bayesian CART and Bayesian neural networks) approaches to
learning from data. In this work, we present two hybrid models combining the
Bayesian and frequentist versions of CART and neural networks, which we call
the Bayesian neural tree (BNT) models. Both models exploit the architecture of
decision trees and have lesser number of parameters to tune than advanced
neural networks. Such models can simultaneously perform feature selection and
prediction, are highly flexible, and generalize well in settings with a limited
number of training observations. We study the consistency of the proposed
models, and derive the optimal value of an important model parameter. We also
provide illustrative examples using a wide variety of real-life regression data
sets.
