Frequently Asked Questions
One-way ANOVA computes a P value <0.05 (significant effect of treatment), but a Tukey multiple comparisons test finds no statistically significant differences between any pairs of means. Is this a contradiction?
FAQ# 782 Last Modified 1-January-2009
This is one of the mysteries of ANOVA. It happens. It is not a software bug.
If one-way ANOVA reports a P value of <0.05, you reject the null hypothesis that all the data are sampled from populations with the same mean. But you cannot be sure that one particular group will have a mean significantly different than another group.
Instead, perhaps the mean of group A and group B differs from the mean of groups C, D and E. Perhaps the mean of group A differs from the mean of groups B through E. Scheffe's post test detects differences like these (not offered by GraphPad Prism or InStat) . If the overall ANOVA P value is less than 0.05, then Scheffe's test will definitely find a significant difference somewhere (if you look at the right comparison, also called contrast).
The Tukey multiple comparisons test only compares pairs of means, and a overall P value < 0.05 in the ANOVA does not guarantee that Tukey (or Dunnett, etc.) multiple comparisons tests will find a significant difference when it compares pairs of mean.
Keywords: posthoc Bonferroni Dunnett post test