Computing a confidence interval for a proportion.

"Exact" confidence intervals are not exactly correct.

Dr. Harvey Motulsky, President GraphPad Software
All contents are copyright © 1995-2002 by GraphPad Software, Inc. All rights reserved.

When an experiment has two possible outcomes, the results are expressed as a proportion. Out of N experiments (or subjects), you observed one outcome (termed "success") in S experiments (or subjects) and the alternative outcome in N-S experiments. Success occurred in S/N of the experiments (or subjects), and we will call that proportion p. Since your data are subject to random sampling, the true proportion of success in the overall population is almost certainly not p. A 95% confidence interval quantifies the uncertainty. You can be 95% sure the overall proportion of success is within the confidence interval.

Prism, InStat and StatMate (and many other programs) compute confidence intervals of proportions using a method developed by Clopper and Pearson (reference 1). Like many statistical texts, the GraphPad manuals and help screens refer to these intervals as being "exact". However, computer simulations by several investigators demonstrate that these intervals are wider than they need to be, and so generally give you more than 95% confidence (references 2 and 3). The discrepancy varies depending on the values of S and N. The so-called "exact" confidence intervals are not, in fact, exactly correct. For all values of S and N, you can be sure that you get at least 95% confidence, but the intervals may be wider than they need to be.

References 2 and 3 review several alternative methods to generate a confidence interval of a proportion. Although none of these methods are perfect – none produce intervals that give you exactly 95% confidence intervals for all possible values of p and N – several give intervals that give you closer to 95% confidence that the so-called "exact" method.

Agresti and Coull (reference 2) recommend a method they term the modified Wald method. It is easy to compute for 95% confidence.

In some cases, the lower limit calculated using that equation is less than zero. In these cases, set the lower limit to 0.0. Similarly, the calculated upper limit can be greater than 1.0. In these cases, set the upper limit to 1.0.

Where did the 2 and 4 come from? You can read the derivation of the equation in reference 2. The numbers 2 and 4 equal z²/2 and z². Since 95% of all values of a normal distribution lie within 1.96 standard deviations of the mean, if you want 95% confidence intervals, set z=1.96 (or to 2 to make it easier to remember). [This paragraph was edited in Feb 2006 to correctly define the value '2' in the equation to equal z²/2 rather than z -- it matters when you compute intervals for confidence levels other than 95%.]

Note that the confidence interval is centered around p’, which is not the same as p, the proportion of experiments that were "successful". Instead p’ is closer to 0.5 than p. This makes sense as the confidence interval can never extend below zero or above one.

Reference 2 shows that this method works very well, as it comes quite close to actually having 95% confidence of containing the true proportion, for any values of S and N. With some values of S and N, the degree of confidence can less than 95%, but it is never has less than 92% confidence.

This table compares the "exact method" (reference 1) used by StatMate, InStat and Prism with the extended Wald method (reference 2) explained above.

With large N, the two methods give very similar results. The extended Wald method gives confidence intervals that are somewhat narrower, which is desirable. The "exact" method actually gives you more than 95% confidence in most cases, which requires wider intervals. With smaller N, the differences are more noticeable, but still unlikely to change your conclusions very much.

Summary

The so-called "exact" method for computing the confidence interval of a proportion (used by Prism, InStat and StatMate) is not, in fact, exact. The intervals tend to wider than the need to be, giving you more than 95% confidence. The discrepancy varies with N and S, but the actual degree of confidence is never lower than 95% for any values of N or S.

There are many alternative ways to compute the confidence interval. None appear to be ideal, and there appears to be no consensus among statisticians for which method is best.

The adjusted Wald method (calculated by Equation) gives narrower confidence intervals (especially with small N) that tend to be quite close to actually giving 95% confidence. With some values of N and S, the actual confidence level can be a bit lower than 95%, but never lower than 92%. Since you want the confidence intervals to be as narrow as possible, you might prefer to use the adjusted Wald method rather than the extended method. However, the differences between the two methods tend to be small.

Because there are several methods to compute a confidence interval of a proportion, you should cite the method you use. StatMate, Prism and InStat use the "exact" method of Clopper and Pearson (reference 1). The adjusted Wald method is detailed in reference 2.

Because the differences between methods tend to be small, and because there appears to be no consensus among statisticians, we don’t plan to change the way that Prism, StatMate or InStat compute the confidence interval of a proportion. Use this free web calculator to compute confidence intervals using the adjusted Wald method.

 References

1. CJ Clopper and ES Pearson, The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26:404-413, 1934.
2. A Agresti and BA Coull, Approximate is better than "Exact" for interval estimation of binomial proportions. The American Statistician. 52:119-126, 1998.
3. RG Newcombe, Two-sided confidence intervals for the single proportion: Comparison of seven methods. Statistics in Medicine 17: 857-872, 1998.

GraphPad Home