  Reviews conducted by the National Academy of Sciences (2009) and the
President's Council of Advisors on Science and Technology (2016) concluded that
the field of forensic firearm comparisons has not been demonstrated to be
scientifically valid. Scientific validity requires adequately designed studies
of firearm examiner performance in terms of accuracy, repeatability, and
reproducibility. Researchers have performed ``black-box'' studies with the goal
of estimating these performance measures. As statisticians with expertise in
experimental design, we conducted a literature search of such studies to date
and then evaluated the design and statistical analysis methods used in each
study. Our conclusion is that all studies in our literature search have
methodological flaws that are so grave that they render the studies invalid,
that is, incapable of establishing scientific validity of the field of firearms
examination. Notably, error rates among firearms examiners, both collectively
and individually, remain unknown. Therefore, statements about the common origin
of bullets or cartridge cases that are based on examination of ``individual"
characteristics do not have a scientific basis. We provide some recommendations
for the design and analysis of future studies.
