  The Infinitesimal Jackknife is a general method for estimating variances of
parametric models, and more recently also for some ensemble methods. In this
paper we extend the Infinitesimal Jackknife to estimate the covariance between
any two models. This can be used to quantify uncertainty for combinations of
models, or to construct test statistics for comparing different models or
ensembles of models fitted using the same training dataset. Specific examples
in this paper use boosted combinations of models like random forests and
M-estimators. We also investigate its application on neural networks and
ensembles of XGBoost models. We illustrate the efficacy of variance estimates
through extensive simulations and its application to the Beijing Housing data,
and demonstrate the theoretical consistency of the Infinitesimal Jackknife
covariance estimate.
