

Nested oneway ANOVA compares the means of three or more unmatched groups, where there is a nested factor within those treatment groups.
Nested ANOVA assumes that the residuals (variation among technical replicates in many cases) are sampled from a Gaussian distribution. This assumption matters less with large samples due to the Central Limit Theorem.
The third tab of the nested t test dialog lets you plot the residuals in several ways to assess their normality.
Nested ANOVA assumes that the data in each subcolumn are sampled from populations with the same SD (same variance). Prism does not test this, but you can look at the data to see if this is badly violated.
Consider running the ANOVA on the logarithms of the values. In some cases this makes the variances much closer to being equal.
Nested oneway ANOVA assumes that variation among subcolumn means is Gaussian and also that the replicates within the subcolumns are Gaussian.
The nested ANOVA compares the means of three or more groups. It is possible to have a tiny P value – clear evidence that the population means are different – even if the distributions overlap considerably. In some situations – for example, assessing the usefulness of a diagnostic test – you may be more interested in the overlap of the distributions than in differences between means.
Prism assumes the groups (data sets) are fixed factors. In other words, Prism tests for differences among the means of the particular groups you have collected data from. It is also possible that group (data sets) represents a random factor. This happens when you have randomly selected groups from an infinite (or at least large) number of possible groups, and that you want to reach conclusions about differences among ALL the groups, even the ones you didn't include in this experiment. Prism cannot perform nested ANOVA when the main factor is random.
Note that Prism does assume that the nested factor is random.
Nested oneway ANOVA asks whether the value of a single variable differs significantly among three or more groups. In Prism, you enter each group in its own column. If the different columns represent different variables, rather than different groups, then oneway ANOVA is not an appropriate analysis. For example, oneway ANOVA would not be helpful if column A was glucose concentration, column B was insulin concentration, and column C was the concentration of glycosylated hemoglobin.