Data Analysis Blog

New in Prism 6: “Exact” P values from multiple comparisons tests

Nearly ever week for the past few years, one or two people have asked how to report “exact” P values from multiple comparisons tests. This seems to stem from instructions to authors for many journals that recommend reporting exact P values rather than inequalities (P<0.05) but don’t distinguish simple statistical results from multiple comparisons tests.

I always wrote back and explained that the whole idea of multiple comparisons is to, well, do multiple comparisons. So the P values don’t stand alone, but rather it makes sense to set one value of alpha for the family of comparisons, and then to sort the comparisons into those that are, and are not, statistically significant by that definition.

In some cases, this turned into a chain of emails. In many cases, it became clear that the scientist didn’t really care what the values meant. She or he just wanted some “exact P values” to stick in the paper and make a reviewer happy. But one of our customers knew a lot about statistics, and pointed me to read about multiplicity adjusted P values. It is a mouthful to say, but turns out to be very useful.

What is a multiplicity adjusted P value? You can read the details, but briefly it is the alpha value (significance threshold) for the entire family of comparisons where the particular comparison you are looking at would just barely be flagged as statistically significant (taking into account multiple comparisons). So if the multiplicity adjusted P value for a particular comparison is 0.03, that means that if you set the significance threshold for the entire family of comparisons to 0.03 (instead of the usual 0.05), that comparison would just barely be considered to be statistically significant.

As their name suggests, multiplicity adjusted P values account for the number of comparisons you are making. For example, if you are comparing three means (A, B, and C), the multiplicity adjusted P value testing the difference between the means of columns A and B would change if you edit the data in column C or add a new data set D to the analysis.

Prism 6 can calculate multiplicity adjusted P values for the Tukey, Dunnett and Bonferroni multiple comparisons tests. This is an option on the third (Options) tab of the new dialogs for one- and two-way ANOVA. Check an option on that tab to make all the settings on that tab become the default for future ANOVAs.

        

Prism 6 also offers a second new (to Prism) method for reporting exact P values after multiple comparisons: Fisher’s unprotected Least Significant Difference (LSD).  This approach, which also reports exact P values for each comparison, does not correct for multiple comparisons, so the P values are lower. Don’t mix up multiplicity adjusted P values with the P values reported by Fishers LSD test.