  Hypothesis tests for the presence of new sources of Poisson counts amidst
background processes are frequently performed in high energy physics (HEP),
gamma ray astronomy (GRA), and other branches of science. While there are
conceptual issues already when the mean rate of background is precisely known,
the issues are even more difficult when the mean background rate has
non-negligible uncertainty. After describing a variety of methods to be found
in the HEP and GRA literature, we consider in detail three classes of
algorithms and evaluate them over a wide range of parameter space, by the
criterion of how close the ensemble-average Type I error rate (rejection of the
background-only hypothesis when it is true) compares with the nominal
significance level given by the algorithm. We recommend wider use of an
algorithm firmly grounded in frequentist tests of the ratio of Poisson means,
although for very low counts the over-coverage can be severe due to the effect
of discreteness. We extend the studies of Cranmer, who found that a popular
Bayesian-frequentist hybrid can undercover severely when taken to high Z
values. We also examine the profile likelihood method, which has long been used
in GRA and HEP; it provides an excellent approximation in much of the parameter
space, as previously studied by Rolke and collaborators.
