  One of the most commonly used methods for forming confidence intervals for
statistical inference is the empirical bootstrap, which is especially expedient
when the limiting distribution of the estimator is unknown. However, despite
its ubiquitous role, its theoretical properties are still not well understood
for non-asymptotically normal estimators. In this paper, under stability
conditions, we establish the limiting distribution of the empirical bootstrap
estimator, derive tight conditions for it to be asymptotically consistent, and
quantify the speed of convergence. Moreover, we propose three alternative ways
to use the bootstrap method to build confidence intervals with coverage
guarantees. Finally, we illustrate the generality and tightness of our results
by a series of examples, including uniform confidence bands, two-sample kernel
tests, minmax stochastic programs and the empirical risk of stacked estimators.
