Three sets of statistical analyses were used to compare the Phase Two results obtained from the purposeful sample of schools (i.e., those with environmental education programs) to Phase One results from the national baseline sample reported by McBeth et al. (2008). The first set of comparisons involved the use of z-tests. In general, z-tests are used to compare the mean for a given sample to the mean of the population from which that sample was drawn in order to determine if there is a significant difference. Because the Phase One sample was a stratified random or probability-proportional sample, and because the Phase One means and standard deviations were weighted to reflect the national population of sixth and eighth graders, those Phase One results are used as population estimates for these two grade levels in z-test
calculations. However, because the Phase One sample included only sixth and eighth graders, only z-tests comparing the Phase One sixth graders to the Phase Two sixth graders and the Phase One eighth graders to the Phase Two eighth graders could be conducted (i.e., no such
comparisons were possible for Phase Two seventh graders). Since separate z-tests were used to compare the Phase One mean scores to Phase Two mean scores on each of the eight parts of the MSELS, the Bonferronni method (Cohen, 1988) was used to adjust the alpha level for this number of statistical comparisons (alpha = .05/8, or .00625).
The results of these comparisons of sixth graders in Phases One and Two are presented in Table 20. When mean scores were compared, the sixth graders in the Phase Two purposeful sample outscored the sixth graders in the Phase One baseline sample on six of the eight parts of the MSELS. The exceptions to this occurred on Part VII.A, Issue Identification, where Phase One sixth graders outscored Phase Two sixth graders, and on Part VII.B, Issue Analysis, where the Phase One and Phase Two mean scores were the same. As a result, these are the only two parts of the MSELS for which z-scores were not greater than zero; the z-score for Issue Identification
Table 20
Z-test and Cohen’s d Comparisons of Phase One and Phase Two Mean Scores for Sixth-Grade Student Samples
Part of MSELS and Variable
(Possible Score) Sample
Sample
*Significant at p<.006125 determined using the Bonferroni method (Cohen, 1988) of dividing the pre-set alpha level (p < .05) by the number of z-tests run (8).
aEffect Size was estimated as Phase Two mean - Phase One mean/ Phase One SD; reported as a percentage of the Phase One SD (e.g., 100% = +1 SD).
was -13.10 and the z-score for Issue Analysis was 0.00. Of these eight comparisons, all z-scores were found to be statistically significant at the adjusted alpha level (p<.006125) except for Part VII.B, Issue Analysis and Part VII.C, Action Planning. It should be noted that the difference on Part VII.A, which favored Phase One, was statistically significant. To summarize, these results indicate that on five of the eight parts of the MSELS, the Phase Two purposeful sample
significantly outscored the Phase One baseline sample, and on one part of the MSELS, the Phase One baseline sample significantly outscored the Phase Two purposeful sample
One of the common criticisms of z-tests and the probability level for resulting z-scores is that these are sensitive to the size of the sample (i.e., the larger the sample size, the greater the chance of finding statistically significant results). In the case of these analyses, the Phase One baseline sample of sixth graders (n = 1,042) and Phase Two purposeful sample of sixth graders (n = 3,134) were sufficiently large for this to be a concern. Further, due to the influence of the sample sizes on these statistical results, mean differences of less than one point were found to be statistically significant on Parts II, V, VI, VII.A, and VII.C of the MSELS. Over time,
researchers and educators have raised questions about the practical or educational significance of results such as these, despite their statistical significance. As a result, statistical approaches were developed to compare results from a practical or educational significance perspective by
removing the influence of sample size. One of these analyses is Cohen’s d (Cohen, 1988).
Cohen’s d is used to calculate the difference between two means, divided by the standard deviation (SD); this often accompanies the reporting of z-test, t-test, and ANOVA results. In simpler terms, Cohen’s d reflects the difference between two means when they are plotted on the same standard distribution curve. The results of this comparison can be represented as a percent of the standard deviation, and are commonly referred to as an effect size. Cohen (1988, p. 25) also defined a small effect size as d >.2 (20% of one SD), a medium effect size as d >.5 (50% of one SD), and a large effect size as d >.8 (80% of one SD).
For the second set of statistical analyses, Cohen’s d was used to compare the Phase One to Phase Two sixth graders on each part of the MSELS. Phase One standard deviation values were used in these calculations to permit the research team to present effect sizes that reflect the relative position of the Phase Two mean above or below the Phase One mean. The results of these
comparisons also are summarized in Table 20. The results indicate that small effect sizes were found on three of the eight parts of the MSELS: two parts in which the Phase Two purposeful sample outscored the Phase One baseline sample (Part IV, Environmental Behavior: d = .263, or 26.3%; and Part VI, Environmental Feelings: d = .23, or 23%), and the part on which the Phase One sample outscored the Phase Two sample (Part VII.A, Issue Identification: d = .24, or 24%).
While the Cohen’s d values for four of the remaining parts of the MSELS were positive, indicating that the Phase Two sample outscored the Phase One sample, these d values fell well below d =.2, reflecting a negligible effect size.