Effect Sizes and P values: different interpretations, both important

Effect sizes quantify the magnitude of differences between groups in ANOVA, providing information beyond what P values alone can tell you. Traditionally, P values have been a central focus of statistical results because they address whether an effect is statistically significant. However, this concept is often misunderstood.

In reality, a "statistically significant" result does not indicate how large or important an effect is. Rather, a P value merely reflects the probability of observing data as extreme as - or more extreme than - the observed data if the null hypothesis were true. For example, a P value of 0.05 means that data like those observed (and thus the effect that was determined) would occur about 5% of the time under the null hypothesis that there is no effect. While this is certainly useful information, it says nothing about the magnitude or practical importance of the effect itself. Instead, it only addresses the question of whether the observed effect is distinguishable from the expected level of random noise in the absence of a true effect.

This is where effect sizes come into play. While a P value addresses whether an effect is detectable relative to random variability, effect sizes describe how large that effect is. A statistically significant result (small P value) can correspond to a very small and practically unimportant effect, particularly with large sample sizes. Conversely, a non-significant result may still reflect a substantial effect that a study simply lacked the power to detect reliably. Effect sizes help you evaluate the practical significance and magnitude of your findings.

Effect Sizes Reported by Prism

Prism reports several different effect size measures depending on the type of ANOVA that you perform and the analysis options you've selected.

•R2 (R squared) - the proportion of variance explained by the model

•Eta squared (η2) - the proportion of variance explained by the associated factor or predictor

•Partial eta squared (Partial η2, ηp2) - the proportion of variance explained by the associated factor/predictor after accounting for other factors or predictors in the model

•Cohen's f - standardized effect expressing explained variance relative to error variance

R squared (R2)

R squared (R2) represents the proportion of total variance in the dependent variable that is explained by entire model (all factors or predictors). Computationally, this effect size is often equivalent to eta squared, though different fields may prefer to report results using one or the other.

R squared in one-way ANOVA

In one-way ANOVA, there is only a single factor (treatment or grouping variable) in the model, so the model contains only one effect. As a result, R squared can be written in a couple of different totally equivalent ways:

It's important to note again that this alternative form only applies to one-way ANOVA. But as a result for these designs, R squared becomes equal mathematically to both η2 and partial η2.

Interpretation

R squared ranges from 0 to 1 (or 0% to 100%). A value of 0.25 means that 25% of the total variance in the data is explained by the model (all factors and interactions). Prism reports R squared for ANOVA in the following places:

•One-way ANOVA - this effect size quantifies the strength of the relationship between group membership and the measured response variable. Since there's only one factor, R squared for the model equals eta squared for that effect

•Repeated measures one-way ANOVA - two R squared values are reported here

1. Treatment effect: here R squared is defined as . This removes the sum of squares for the individual/subjects from equation, focusing on the group membership or treatment effect variance relative to random error. In this case, the calculation is equivalent to partial eta squared

2. Effectiveness of matching: here R squared is defined as . This quantifies what proportion of total variance is due to consistent differences between subjects.

Eta squared (η2)

Eta squared quantifies the proportion of total variance attributable to a specific individual effect in ANOVA (such as one of the main effects/factors or an interaction term). Unlike R2 which describes the entire model, eta squared is calculated separately for each effect in the analysis.

For any specific effect in ANOVA:

Values of eta squared can range from 0 to 1 (0% to 100%), indicating what portion of total variance the specific effect explains.

Relationship to R squared in one-way ANOVA

In one-way ANOVA, there is only one effect (the grouping factor), so:

•Eta squared for the main effect is equal to R squared for the model

•These values are numerically identical because there's only one effect to evaluate

•The interpretation of these effect sizes are still subtly different even if their numeric values are the same

In factorial designs (two-way, three-way, or multifactor ANOVA), each effect has its own eta squared value which are different from an overall model R squared.

Partial eta squared (partial η2)

Partial eta squared is commonly reported in factorial ANOVA designs (two-way, three-way, or multifactor ANOVA). Unlike eta squared which uses the total variance as the denominator, partial eta squared uses only the variance relevant to the effect being examined.

For any specific effect in ANOVA:

The denominator of this formula provides insight into the question that partial eta squared answers. The total sum of squares is equal to the sum of squares for each component (effects and interactions) of the ANOVA model plus the error or residual sum of squares. Thus, if eta squared answers the question "what proportion of the total variance is explained by this effect?", then partial eta squared answers the related question "what proportion of the variance unexplained by other effects is explained by this effect?". In other words, partial eta squared provides an effect size for each effect of an ANOVA model after first accounting for all of the other effects in the model.

This makes partial eta squared particularly useful for:

•Comparisons of effect sizes across different studies with similar designs

•Understanding the unique contribution of each factor

•Factorial designs where you want to assess each factor independently

Values for partial eta squared can range from 0 to 1 (0% to 100%), indicating what proportion of variance unexplained by other factors in the model this specific factor explains.

Cohen's f

Cohen's f is another measure of effect size that is particularly useful for power analysis and sample size calculations. For any given effect, Cohen's f comes from the following related definition:

This effectively makes the quantity f2 a signal-to-noise ratio. Previously, we showed that partial eta squared was the proportion of variance explained by the effect after removing the contribution of other effects. Since we know that the contribution of other effects has been removed, the only remaining relevant sources of variance must be that explained by the effect and the error variance. Thus, if ηp2 is the variance explained by the effect, then 1-ηp2 must be the error variance! Thus, we arrive at:

Partial eta squared tells us what fraction of the relevant variation is due to an effect. Cohen's f simply expresses that same information as a signal-to-noise ratio rather than a proportion. Because of this mathematical conversion, Cohen's f is not constrained to a value between 0 and 1.

Often, you'll see assessments of Cohen's f based on some standard values:

It's very important to keep in mind that these interpretation guidelines are useful in many cases, but what's considered a "small" "medium" or "large" effect may vary dramatically from field to field and even from experiment to experiment. These standard guidelines cannot serve as a total replacement for knowledge and understanding of the research area that you're involved in.