  Decision tree ensembles are widely used and competitive learning models.
Despite their success, popular toolkits for learning tree ensembles have
limited modeling capabilities. For instance, these toolkits support a limited
number of loss functions and are restricted to single task learning. We propose
a flexible framework for learning tree ensembles, which goes beyond existing
toolkits to support arbitrary loss functions, missing responses, and multi-task
learning. Our framework builds on differentiable (a.k.a. soft) tree ensembles,
which can be trained using first-order methods. However, unlike classical
trees, differentiable trees are difficult to scale. We therefore propose a
novel tensor-based formulation of differentiable trees that allows for
efficient vectorization on GPUs. We perform experiments on a collection of 28
real open-source and proprietary datasets, which demonstrate that our framework
can lead to 100x more compact and 23% more expressive tree ensembles than those
by popular toolkits.
