  We present a robust deep incremental learning framework for regression tasks
on financial temporal tabular datasets which is built upon the incremental use
of commonly available tabular and time series prediction models to adapt to
distributional shifts typical of financial datasets. The framework uses a
simple basic building block (decision trees) to build self-similar models of
any required complexity to deliver robust performance under adverse situations
such as regime changes, fat-tailed distributions, and low signal-to-noise
ratios. As a detailed study, we demonstrate our scheme using XGBoost models
trained on the Numerai dataset and show that a two layer deep ensemble of
XGBoost models over different model snapshots delivers high quality predictions
under different market regimes. We also show that the performance of XGBoost
models with different number of boosting rounds in three scenarios (small,
standard and large) is monotonically increasing with respect to model size and
converges towards the generalisation upper bound. We also evaluate the
robustness of the model under variability of different hyperparameters, such as
model complexity and data sampling settings. Our model has low hardware
requirements as no specialised neural architectures are used and each base
model can be independently trained in parallel.
