  With the advent of high-throughput screenings, it has become increasingly
common for studies to devote limited resources to estimating many parameters
imprecisely rather than to estimating a few parameters well. In these studies,
only two or three independent replicates measure each parameter, and therefore
it is challenging to assess the variance of these measurements. One solution is
to pool variance estimates across different parameters using a parametric model
of estimator error. However, such models are difficult to specify correctly,
especially in the presence of ``batch effects.'' In this paper, we propose new
model-free methods for assessing and controlling estimator error. Our focus is
on type S error, which is of particular importance in many settings. To produce
tight confidence intervals without making unrealistic assumptions, we improve
on Hoeffding's bounds for sums of bounded random variables and obtain the
tightest possible Chernoff-Cram\'er bound. Our methods compare favorably with
existing practice for high-throughput screenings, such as methods based on the
Irreproducible Discovery Rate (IDR) and the Benjamini-Hochberg procedure.
Existing practices fail to control error at the nominal level in some cases and
are needlessly conservative in others.
