Go to the GraphPad home page. Buy online now for instant delivery!

  GraphPad Home Support Quick Answers Database

How can I get exact P values for the post tests that follow one-way or two-way ANOVA?   FAQ# 189
  

This sounds like a simple question, but in fact it can only be answered by discussing some fundamental, almost philosophical, questions.

The distinction between P values and statistical hypothesis testing

When you make one comparison, it is easy to gloss over the difference between reporting a P value and using statistical hypothesis testing to report a conclusion of whether or not that difference is "statistically significant". But the two are somewhat distinct:

  • A P value is a measure of strength-of-evidence -- of how well (or how poorly) your data comply with the null hypothesis.
  • A statement of 'significance' is a way to divide all your tests into two piles -- ones that you think are worthy of follow up ('significant'), and those that are not.

Multiple comparisons applies to a set of comparisons, and not to any one particular comparison

When you make several comparisons at once, there is a huge distinction between P values and statistical hypothesis testing. The methods of statistical hypothesis testing can be adjusted to account for the fact you are making multiple comparisons at one time. When you make one comparison, you usually set alpha to 5% which means: If the null hypothesis is true, there is a 5% chance of ending up with a 'statistically significant' result just by chance. When you are making multiple comparisons, you can set things up so that 5% value applies to the entire family of comparisons. The 5% is said to be the family-wise error rate. This means that if all the null hypotheses were true, there is a 5% chance that one or more of the differences will be 'statistically significant' simply due to random variation, leaving a 95% chance that all the comparisons will be 'not significant'.

Reporting an individual P value, by its very nature, is a way to express the strength-of-evidence for that one comparison. It doesn't really make a lot of sense to compute individual P values (but see below)

Prism does report the results at several different significance levels (0.05, 0.01, 0.001). 

Multiplicity adjusted P values

Some have proposed adjusting the P value for multiple comparisons. A P value is calculated for each comparison, taking into account the others. The trick is knowing how to interpret these P values. Each multiplicity adjusted P value is the significance level at which that particular comparison would just barely be considered statistically significant. More on adjusted P values. 

t tests after one-way ANOVA, without correcting for multiple comparisons

An alternative approach is to compute a P value for each comparison, without adjusting for multiple comparisons. This is sometimes called the unprotected Fisher's Least Significant Difference test. More

 

Keywords: Tukey, newman keuls, posthoc, Dunnett, exact P value, multiple comparison
* * * * *
Copyright 2002 - 2009, GraphPad Software Inc. All rights reserved.

Please email us comments and corrections.