  This work develops formal statistical inference procedures for machine
learning ensemble methods. Ensemble methods based on bootstrapping, such as
bagging and random forests, have improved the predictive accuracy of individual
trees, but fail to provide a framework in which distributional results can be
easily determined. Instead of aggregating full bootstrap samples, we consider
predicting by averaging over trees built on subsamples of the training set and
demonstrate that the resulting estimator takes the form of a U-statistic. As
such, predictions for individual feature vectors are asymptotically normal,
allowing for confidence intervals to accompany predictions. In practice, a
subset of subsamples is used for computational speed; here our estimators take
the form of incomplete U-statistics and equivalent results are derived. We
further demonstrate that this setup provides a framework for testing the
significance of features. Moreover, the internal estimation method we develop
allows us to estimate the variance parameters and perform these inference
procedures at no additional computational cost. Simulations and illustrations
on a real dataset are provided.
