  The best-performing models in ML are not interpretable. If we can explain why
they outperform, we may be able to replicate these mechanisms and obtain both
interpretability and performance. One example are decision trees and their
descendent gradient boosting machines (GBMs). These perform well in the
presence of complex interactions, with tree depth governing the order of
interactions. However, interactions cannot fully account for the depth of trees
found in practice. We confirm 5 alternative hypotheses about the role of tree
depth in performance in the absence of true interactions, and present results
from experiments on a battery of datasets. Part of the success of tree models
is due to their robustness to various forms of mis-specification. We present
two methods for robust generalized linear models (GLMs) addressing the
composite and mixed response scenarios.
