Data Analysis Blog

Statistical significance defined using the five sigma standard

Many fields of science (biology, clinical trials, psychology) share the same definition of "statistical signifiance":  P < 0.05.  

Not partical physics! When a large international group of physics recently announced the discovery of the Higgs Boson, they used a much stricter threshold. The results were not announced in terms of a P value or statistical significance. Instead they announced that two sperate data sets meet the five-sigma threshold.

What does "five sigma" mean? It means that the results would occur by chance alone as rarely as a value sampled from a Gaussian distribution would be five standard deviations from the mean. The Excel function 1- normsdist(5) computes that the fraction of the normal distribution that is greater than five standard deviaitons from the mean. The value is 0.0000003, or 0.00003%, or about one in three and a half million. In other words, the one-tailed P value is less than  0.0000003. If the Higgs Boson doesn't exist, that is the chance that coincidences would align to give results as striking as they observed.  And they obtained this level of evidence in two independent sets of experiments.

The standard of statistical signifcance in most fileds is that the two-tailed P is less than 0.05. About five percent of a Gaussian distribution is more than two standard deviations away from the mean (in either direction). So the conventional definition of statistical significance can be called a two-sigma threshold. 

The tradition in particle physics is that the threshold to report “evidence of a particle,” is p<0.003 (three sigma), and the standard to report a “discovery” is p<0.0000003 (five sigma).

My goal in this blog is to simply point out that different fields use different thresholds. The P<0.05 standard is not universal.