  Though introduced nearly 50 years ago, the infinitesimal jackknife (IJ)
remains a popular modern tool for quantifying predictive uncertainty in complex
estimation settings. In particular, when supervised learning ensembles are
constructed via bootstrap samples, recent work demonstrated that the IJ
estimate of variance is particularly convenient and useful. However, despite
the algebraic simplicity of its final form, its derivation is rather complex.
As a result, studies clarifying the intuition behind the estimator or
rigorously investigating its properties have been severely lacking. This work
aims to take a step forward on both fronts. We demonstrate that surprisingly,
the exact form of the IJ estimator can be obtained via a straightforward linear
regression of the individual bootstrap estimates on their respective weights or
via the classical jackknife. The latter realization is particularly useful as
it allows us to formally investigate the bias of the IJ variance estimator and
better characterize the settings in which its use is appropriate. Finally, we
extend these results to the case of U-statistics where base models are
constructed via subsampling rather than bootstrapping and provide a consistent
estimate of the resulting variance.
