What is the best technique to calculate a confidence interval of a binomial experiment, if your estimate is that $p=0$ (or similarly $p=1$) and sample size is relatively small, for example $n=25$?
Do not use the normal approximation

Much has been written about this problem. A general advice is to never use the normal approximation (i.e., the asymptotic/Wald confidence interval), as it has terrible coverage properties. R code for illustrating this:

library(binom)
p = seq(0,1,.001)
coverage = binom.coverage(p, 25, method="asymptotic")$coverage
plot(p, coverage, type="l")
binom.confint(0,25)
abline(h=.95, col="red")




For small success probabilities, you might ask for a 95% confidence interval, but actually get, say, a 10% confidence interval!

Recommendations

So what should we use? I believe the current recommendations are the ones listed in the paper Interval Estimation for a Binomial Proportion by Brown, Cai and DasGupta in Statistical Science 2001, vol. 16, no. 2, pages 101–133. The authors examined several methods for calculating confidence intervals, and came to the following conclusion.


  [W]e recommend the Wilson interval or the equal-tailed Jeffreys prior interval for small n and the interval suggested in Agresti and Coull for larger n.


The Wilson interval is also sometimes called the score interval, since it’s based on inverting a score test.

Calculating the intervals

To calculate these confidence intervals, you can use this online calculator or the binom.confint() function in the binom package in R. For example, for 0 successes in 25 trials, the R code would be:

> binom.confint(0, 25, method=c("wilson", "bayes", "agresti-coull"),
  type="central")
         method x  n  mean  lower upper
1 agresti-coull 0 25 0.000 -0.024 0.158
2         bayes 0 25 0.019  0.000 0.073
3        wilson 0 25 0.000  0.000 0.133


Here bayes is the Jeffreys interval. (The argument type="central" is needed to get the equal-tailed interval.)

Note that you should decide on which of the three methods you want to use before calculating the interval. Looking at all three and selecting the shortest will naturally give you too small coverage probability.

A quick, approximate answer

As a final note, if you observe exactly zero successes in your n trials and just want a very quick approximate confidence interval, you can use the rule of three. Simply divide the number 3 by n. In the above example n is 25, so the upper bound is 3/25 = 0.12 (the lower bound is of course 0).
