Every P value Prism reports for a fixed effect in a mixed effects model comes from comparing a test statistic to an F or t distribution. The numerator degrees of freedom are easy to work out, because they simply count the parameters being tested. The denominator degrees of freedom are not obvious at all. Prism gives you three ways to compute them.

Why the denominator degrees of freedom are a problem

Ordinary ANOVA partitions the total variation into strata. Each F test has an obvious error term, that error term has an obvious number of degrees of freedom, and all the pieces add up to the total. There is nothing to decide.

A mixed model does not work that way. It estimates the fixed effects by generalized least squares using a covariance matrix that is itself estimated from your data by restricted maximum likelihood (REML). The variance of a fixed effect estimate is therefore an estimate resting on top of another estimate, and a test statistic built from it does not follow an F distribution exactly. It follows one only approximately, and how good that approximation is depends on how much uncertainty there is in the estimated variance components.

When you have many subjects, that uncertainty is negligible and any sensible choice of denominator degrees of freedom gives nearly the same P value. When you have few subjects, an unbalanced design, or missing values, the choice matters. Two methods can report noticeably different P values from exactly the same fitted model.

Note that the fit itself does not change when you switch methods. The REML fit, the fixed effect estimates, the predicted means, and the random effect SDs all stay the same. What changes is the reference distribution used to turn a test statistic into a P value.

The three choices

•Legacy. The method used by earlier versions of Prism. It hands each test the degrees of freedom of the matching ANOVA error term, worked out from the design. It costs nothing additional to compute and it is exactly right when your data are complete and balanced. It is not reliable otherwise.

•Satterthwaite. Estimates an effective denominator degrees of freedom for each test from the uncertainty in the estimated variance components. It is more complex computation than Legacy though much less than Kenward-Roger, and it is a large improvement over Legacy on incomplete or unbalanced data.

•Kenward-Roger. Does everything Satterthwaite does, and also corrects the estimated covariance matrix of the fixed effects for the bias introduced by estimating the variance components. It is the most accurate of the three and the most complex in terms of computation. The primary downside is that for some designs, this method may be slower than the others. This is the default method in Prism.

The two new methods each have their own page: Kenward-Roger degrees of freedom and Satterthwaite degrees of freedom. The Legacy method is described below.

How the Legacy method works

The Legacy method treats the mixed model as though it were the corresponding repeated measures ANOVA. It builds the ANOVA error strata from the design, counts the degrees of freedom in each one, and gives every fixed effect the count belonging to its matching error term.

Every one of those counts comes from the design rather than from the data. With s subjects and a treatment levels, the Subject × Treatment term always carries (s − 1)(a − 1) degrees of freedom. The nested random factor term always carries the number of subcolumns minus the number of groups. The within cells term is whatever is left over once the fixed effect parameters and all the subject strata are removed:

where n is the number of values actually analyzed and p is the number of fixed effect parameters.

Why Legacy is reliable only for complete, balanced data

Follow what happens when one value goes missing. The count n drops by one, so the within cells term loses a degree of freedom. Every subject stratum count is a product of level counts, so not one of them changes.

The within cells term therefore absorbs the entire loss of information, while every other error term keeps degrees of freedom for observations that are not there. Any test whose denominator comes from a Subject × factor stratum, which is most of the tests in a three-way repeated measures analysis, is credited with more information than your data actually contain. Its P value comes out too small.

That is why Prism warns you when an analysis with missing values uses the Legacy method, and why Legacy is a poor choice for any analysis that is not both complete and balanced.

Comparing the three methods

When your data are complete and balanced, all three methods give the same answer, and that answer matches ordinary repeated measures ANOVA.

Which method should you choose?

Kenward-Roger is the default, and it is the right choice for almost every analysis. Choose it when:

•The number of subjects, animals, litters, plates, or blocks is small or moderate.

•The design is unbalanced.

•Any values are missing.

•You are preparing results for publication or for a regulated submission.

•You want the closest possible agreement with SAS or with the R package pbkrtest.

Choose Satterthwaite when:

•Your model is large enough that Kenward-Roger takes an uncomfortably long time.

•You want to match an analysis a colleague ran in R with lmerTest or emmeans.

Choose Legacy when:

•You are reproducing results computed by an earlier version of Prism.

•Your data are complete and balanced, in which case all three methods agree anyway.

If you have a very large number of subjects, the three methods converge and the choice starts to matter less.

What changes in your results

•The ANOVA table. Denominator degrees of freedom and P values change. With Kenward-Roger the F ratios change too, because the standard errors they are built from are adjusted.

•Fractional degrees of freedom. Satterthwaite and Kenward-Roger both report degrees of freedom that are usually not whole numbers. This is expected and is not a sign of a problem.

•Multiple comparisons. Each comparison gets its own degrees of freedom, so different rows of the same table can show different values. Adjusted P values and confidence intervals change along with them.

•Nothing about the fit. The estimated means, the fixed effect estimates, the random effect SDs and variances, the test of whether matching was effective, and the goodness of fit measures are all unchanged.

Satterthwaite and Kenward-Roger can both be used with the multiple comparisons methods that rest on t-based inference: Bonferroni, Šídák, Holm-Šídák, the FDR methods, and uncorrected Fisher's LSD.

When a random effect has an estimated SD of zero

Satterthwaite and Kenward-Roger both need a complete set of estimated variance components and a usable covariance matrix for the fixed effects. If a random effect is estimated to have an SD of zero and you have chosen Remove term(s) from model and fit a simpler model, that term is gone from the model and the derivatives these methods need cannot be formed. Prism falls back to the Legacy method for that result and posts a floating note telling you so.

To keep your chosen method available, select Restrict the variance to zero and keep the term(s) instead. The term stays in the model with its variance pinned at zero, which leaves the covariance structure intact.

Where the choice appears

The Define denominator degrees of freedom (mixed-effects model) section is on the Repeated Measures tab of the one-way, two-way, and three-way ANOVA parameters dialogs, and on the Analysis tab of the nested one-way ANOVA parameters dialog. It is available only when both of the following are true:

•The analysis fits a mixed effects model by REML.

•Sphericity is assumed.

The section is available only when you choose the mixed-effects model. It is unavailable for repeated measures ANOVA based on the general linear model, which is the only case where you can choose the Geisser-Greenhouse correction (in the sphericity section of the Repeated Measures tab). The Geisser-Greenhouse correction also adjusts degrees of freedom, but it does so for a different reason and on a different analysis path.

Opening the file in an earlier version of Prism

Versions of Prism that do not offer these methods cannot recalculate an analysis that uses one. The saved results stay visible but frozen, so you can read them but not recalculate them or edit the analysis parameters.

These analyses also cannot be saved in the older .pzfx and .pzf file formats. The Compatibility Checker treats this as a critical problem and names the sheets involved.

A file saved by an earlier version of Prism carries no record of which method was used, so Prism opens those analyses with the Legacy method.