  We formulate the problem of constructing multiple simultaneously valid
confidence intervals (CIs) as estimating a high probability upper bound on the
maximum error for a class/set of estimate-estimand-error tuples, and refer to
this as the error estimation problem. For a single such tuple, data-driven
confidence intervals can often be used to bound the error in our estimate.
However, for a class of estimate-estimand-error tuples, nontrivial high
probability upper bounds on the maximum error often require class complexity as
input -- limiting the practicality of such methods and often resulting in loose
bounds. Rather than deriving theoretical class complexity-based bounds, we
propose a completely data-driven approach to estimate an upper bound on the
maximum error. The simple and general nature of our solution to this
fundamental challenge lends itself to several applications including: multiple
CI construction, multiple hypothesis testing, estimating excess risk bounds (a
fundamental measure of uncertainty in machine learning) for any
training/fine-tuning algorithm, and enabling the development of a contextual
bandit pipeline that can leverage any reward model estimation procedure as
input (without additional mathematical analysis).
