  We propose a new Bayesian model for flexible nonlinear regression and
classification using tree ensembles. The model is based on the RuleFit approach
in Friedman and Popescu (2008) where rules from decision trees and linear terms
are used in a L1-regularized regression. We modify RuleFit by replacing the
L1-regularization by a horseshoe prior, which is well known to give aggressive
shrinkage of noise predictor while leaving the important signal essentially
untouched. This is especially important when a large number of rules are used
as predictors as many of them only contribute noise. Our horseshoe prior has an
additional hierarchical layer that applies more shrinkage a priori to rules
with a large number of splits, and to rules that are only satisfied by a few
observations. The aggressive noise shrinkage of our prior also makes it
possible to complement the rules from boosting in Friedman and Popescu (2008)
with an additional set of trees from random forest, which brings a desirable
diversity to the ensemble. We sample from the posterior distribution using a
very efficient and easily implemented Gibbs sampler. The new model is shown to
outperform state-of-the-art methods like RuleFit, BART and random forest on 16
datasets. The model and its interpretation is demonstrated on the well known
Boston housing data, and on gene expression data for cancer classification. The
posterior sampling, prediction and graphical tools for interpreting the model
results are implemented in a publicly available R package.
