  Gradient boosted trees are competition-winning, general-purpose,
non-parametric regressors, which exploit sequential model fitting and gradient
descent to minimize a specific loss function. The most popular implementations
are tailored to univariate regression and classification tasks, precluding the
possibility of capturing multivariate target cross-correlations and applying
structured penalties to the predictions. In this paper, we present a
computationally efficient algorithm for fitting multivariate boosted trees. We
show that multivariate trees can outperform their univariate counterpart when
the predictions are correlated. Furthermore, the algorithm allows to
arbitrarily regularize the predictions, so that properties like smoothness,
consistency and functional relations can be enforced. We present applications
and numerical results related to forecasting and control.
