  Gradient Boosted Decision Trees (GBDT) is a very successful ensemble learning
algorithm widely used across a variety of applications. Recently, several
variants of GBDT training algorithms and implementations have been designed and
heavily optimized in some very popular open sourced toolkits including XGBoost,
LightGBM and CatBoost. In this paper, we show that both the accuracy and
efficiency of GBDT can be further enhanced by using more complex base learners.
Specifically, we extend gradient boosting to use piecewise linear regression
trees (PL Trees), instead of piecewise constant regression trees, as base
learners. We show that PL Trees can accelerate convergence of GBDT and improve
the accuracy. We also propose some optimization tricks to substantially reduce
the training time of PL Trees, with little sacrifice of accuracy. Moreover, we
propose several implementation techniques to speedup our algorithm on modern
computer architectures with powerful Single Instruction Multiple Data (SIMD)
parallelism. The experimental results show that GBDT with PL Trees can provide
very competitive testing accuracy with comparable or less training time.
