  Forensic science plays a critical role in the United States criminal justice
system. For decades, many feature-based fields of forensic science, such as
firearm and toolmark identification, developed outside the scientific
community's purview. The results of these studies are widely relied on by
judges nationwide. However, this reliance is misplaced. Black-box studies to
date suffer from inappropriate sampling methods and high rates of missingness.
Current black-box studies ignore both problems in arriving at the error rate
estimates presented to courts. We explore the impact of each type of limitation
using available data from black-box studies and court materials. We show that
black-box studies rely on non-representative samples of examiners. Using a case
study of a popular ballistics study, we find evidence that these
unrepresentative samples may commit fewer errors than the wider population from
which they came. We also find evidence that the missingness in black-box
studies is non-ignorable. Using data from a recent latent print study, we show
that ignoring this missingness likely results in systematic underestimates of
error rates. Finally, we offer concrete steps to overcome these limitations.
