﻿ Exact P values from multiple comparisons tests

# Exact P values from multiple comparisons tests

Prism reports exact P values from multiple comparisons tests using two different approaches that are quite different.

Prism can compute a multiplicity adjusted P value for each comparison for many multiple comparison methds: The methods of Tukey, Dunnett, Bonferroni, Sidak, Dunn, and Holm. Since adjusted P values are not reported by most programs, and are not widely reported in scientific papers (yet), be sure you fully understand what they mean before reporting these values.

A multiplicity adjusted P value  is the family-wise significance level at which that particular comparison would just barely be considered statistically significant. That is a hard concept to grasp. You can set the threshold of significance, for the whole family of comparisons, to any value you want.  Usually, it is set to 0.05 or 0.01 or perhaps 0.10. But it can be set to any value you want, perhaps 0.0345. The adjusted P value is the smallest significance threshold, for the entire family of comparisons, at which this one comparison would be declared "statistically significant".

The adjusted P value for each comparison depends on all the data, not just the data in the two groups that P value compares. If you added one more comparison to the study (or took one away), all the adjusted P values would change. The adjusted P value can be thought of as a measure of the strength of evidence.

## P values that don't correct for multiple comparisons

Fisher's Least Significant Difference (LSD) test computes a P value (and confidence interval) for each comparison, without adjusting for multiple comparisons. The results will be similar to performing independent t tests for each comparison, except the Fishers LSD test uses all the data to compute a pooled standard deviation (rather than using the variation only in the two groups being compared). This will usually give it more power than independent t tests. When reporting P values from the Fishers LSD test, be sure to explain that these do not account for multiple comparisons, the reader must do so when evaluating the results.

The uncorrected Dunn's test is the nonparametric test that computes a P value for each comparison without correcting for multiple comparisons.

## Adjusted P values are very different than P values that don't account for multiple comparisons

Multiplicity adjusted P values, as the name suggests, accounts for multiple comparisons.

The Fisher LSD test and the uncorrected Dunn's test (nonparametric) do not account for multiple comparisons.

The "exact" P values computed by the two approaches are not the same. If you report either, be sure to be very explicit about exactly what P value you are reporting.