6. Statistical comparisons: Items with mean differences that are larger than would be expected by chance are noted with asterisks referring to three significance levels (*p <.05, **p <.01, ***p <.001). Significance levels indicate the probability that an observed difference is due to chance. Statistical significance does not guarantee the result is substantive or important. Large sample sizes tend to generate more statistically significant results even though the magnitude of mean differences may be inconsequential. Consult effect sizes (see #7) to judge the practical meaning of differences. Unless otherwise noted, statistical comparisons are two-tailed independent t-tests. Exceptions are items 11 a-f which are compared using a z-test.

7. Effect size: Effect size indicates practical significance. An effect size of .2 is often considered small, .5 moderate, and .8 large. A positive effect size indicates that your institution's mean was greater than that of the comparison group, thus showing a favorable result for your institution. A negative effect size indicates your institution lags behind the comparison group, suggesting that the student behavior or institutional practice represented by the item may warrant attention. Effect sizes for independent t-tests use Cohen's d; z-tests use Cohen's h. Cohen's d is calculated by dividing the mean difference by the pooled standard deviation. Cohen's h is calculated by taking the difference in the proportion of students who responded "Done or in progress" after the proportion has been transformed using a non-linear (arcsine) transformation. See: Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd edition). New York: Psychology Press.