Dunn's test that corrects for multiple comparisons

Prism performs the Dunn's multiple comparison test(1), which is standard. One source is Daniel's Applied nonparametric statistics, second edition page 240-241. Some books and programs don't use Dunn's name, but simply refer to this test as the post test following a Kruskal-Wallis test, and don't give it an exact name.

1.Compute the value of alpha that accounts for multiple comparisons. Divide the family significance threshold,usually  0.05, by the number of comparisons you are making. If you compare each group with every other group, you are making K*(K-1)/2 comparisons, where K is the number of groups. If you are comparing each group to a control group, you are making K-1 comparisons. If you only comparing a set of preselected pairs of groups or treatments, K is the number of comparisons.

2.Find the value of z from the normal distribution that corresponds to that two-tailed probability. This free calculator will help. For example, if there are 4 groups, you are making 6 comparisons, and the critical value of z (using the usual 0.05 significance level for the entire family of comparisons) is the z ratio that corresponds to a probability of 0.05/6 or 0.008333. That z ratio is  2.638.

3.The next step differs for ordinary and repeated measures ANOVA.

For ordinary (not matched, not repeated measures) nonparametric ANOVA: To compare group i and j, find the absolute value of the difference between the mean rank of group i and the mean rank of group j. If there are no ties, compute z by dividing this difference in mean ranks by the square root of [(N*(N+1)/12)*(1/Ni + 1/Nj)]. Here N is the total number of data points in all groups, and Ni and Nj are the number of data points in the two groups being compared. If there are ties, calculate z by dividing the difference in mean ranks by the square root of [(N*(N+1) - Sum(Ti^3 - Ti) / (N - 1)) / 12 * (1/Ni + 1/Nj), where Ti is the number of ties in the i-th group of ties.

For repeated measures nonparametric ANOVA (Friedman's test): To compare treatment i and j, find the absolute value of the difference between the mean rank of group i and the mean rank of group j. Calculate z by dividing this difference in mean ranks by the square root of [K(K+1)/(6N)]. Here N is the number of matched sets of data, which is the number of rows in the data table, and K is the number of treatment groups (number of columns).

4.If the z ratio calculated in the preceding paragraph is larger that the critical value of z computed in the paragraph before that, then conclude that the difference is statistically significant.

Uncorrected Dunn's test (does not correct for multiple comparisons)

1.The details differ for ordinary and repeated measures ANOVA.

For ordinary (not matched, not repeated measures) nonparametric ANOVA: To compare group i and j, find the absolute value of the difference between the mean rank of group i and the mean rank of group j. If there are no ties, compute z by dividing this difference in mean ranks by the square root of [(N*(N+1)/12)*(1/Ni + 1/Nj)]. Here N is the total number of data points in all groups, and Ni and Nj are the number of data points in the two groups being compared. If there are ties, calculate z by dividing the difference in mean ranks by the square root of [(N*(N+1) - Sum(Ti^3 - Ti) / (N - 1)) / 12 * (1/Ni + 1/Nj), where Ti is the number of ties in the i-th group of ties.

For repeated measures nonparametric ANOVA (Friedman's test): To compare treatment i and j, find the absolute value of the difference between the mean rank of group i and the mean rank of group j. Calculate z by dividing this difference in mean ranks by the square root of [K(K+1)/(6N)]. Here N is the number of matched sets of data, which is the number of rows in the data table, and K is the number of treatment groups (number of columns).

2.Compute and report the two-tailed P value that corresponds to the computed z ratio. When comparing two particular groups, each rank is determined from the entire data set (all groups), not just the two groups being compared. But only the two sum of ranks from the two groups being compared enter into the calculation of z.

This is the same method used in the R method Dunn.test if you choose one of the FDR methods to control for multiple comparisons.

Notes:

•When comparing two groups as part of Dunn's test, the ranks are those for all the values. Rank all the values, compute the mean rank in each group, and do the calculations above. Don't create ranks just for the values in those two groups.

• This method accounts for ties when computing the ranks, and thus when computing the mean ranks which are compared.

•It would seem sensible to base multiple comparisons tests after nonparametric ANOVA to be based on the Mann-Whitney or Wilcoxon method to compute P values, followed by Bonferroni or other method to correct for multiple comparisons. Prism does not offer this approach, because it is not commonly used (but we'd appreciate comments and suggestions).

1. O.J. Dunn, Technometrics, 5:241-252, 1964