Salud

CMOs that tend to adopt one practice often adopt other practices that are consistent with their overall educational strategy. For example, as noted above, some CMOs that provide intensive coaching also employ formative assessments frequently to help teachers determine which skills or topics to prioritize. In addition to correlating pairs of practices, we used cluster analysis to explore whether CMOs can be categorized based on the mix of practices they employ; examining whether CMOs bundle certain practices may provide insight into their strategic approach. Below we discuss two ways to categorize CMOs, based on (1) the prescriptiveness of their policies and (2) the core practices that define our primary hypotheses regarding the drivers of impacts.17

1. Four groups of CMOs based on extent and form of CMO prescriptiveness

We explored the extent to which CMOs centralize decision-making or delegate authority to school principals using several items from the principal survey. Specifically, we used responses to survey questions that address whether CMO or school staff typically make key decisions relating to three broad areas: (1) curriculum/instructional approach, assessment, and professional development; (2) teacher evaluation and compensation; and (3) behavior policy.18 _{With respect to these three} broad dimensions of centralization, cluster analysis indicates that CMOs fall into four categories.19

17 _{We conducted a hierarchical cluster procedure in SAS using Ward’s minimum variance method, in which the}

distance between clusters is defined as the ANOVA sum of squares between two clusters, summed over the variables specified. At each step in the procedure, clusters from the previous step are merged so as to minimize the within-cluster sum of squares. Although we were interested in the potential grouping of CMOs on dimensions of prescriptiveness and school-level practices, we found no pattern of correlations of our defined core practices with prescriptiveness. Therefore, we explored CMO centralization as a separate domain for the purposes of our cluster analysis.

Group A CMOs tend to be very centralized in decision making only about policies and practices related to instruction (that is, educational approach, professional development, and formative assessments). Group B CMOs tend to be highly prescriptive or centralized in decision-making across all dimensions measured. Group C CMOs are very prescriptive with regard to behavior policy

18 _{We created five composite measures of prescriptiveness using principal survey items pertaining to decisions}

made or resources provided by the CMO to its schools: (1) CMO-prescribed educational approach, (2)centralized provision of professional development, (3) centralized policy on teacher evaluation and compensation, (4) CMO- prescribed formative assessment system, and (5) centralized behavior policy. From these composite measures, we developed three vectors of prescriptiveness or centralization using principal components analysis: (1) prescriptiveness on instructional policies, (2) centralized policy on teacher evaluation and compensation, and (3) centralized behavior policy.

and decentralized elsewhere, on average. Group D CMOs appear to be very decentralized in decision making across all dimensions, on average.

These group typologies are depicted in Table III.4. Each row indicates one of the dimensions of principal-reported prescriptiveness, while the mean values and corresponding rankings across these dimensions are reported by the group in each column. The CMO and district comparison means are provided for reference. The column for Group A, for example, illustrates that those CMOs tend to be highly prescriptive and rank first in their degree of centralization of teacher evaluation criteria and compensation on average, while they are substantially less apt to mandate practices in the other areas measured.

Table III.4. Prescriptiveness Varies Across CMOs and Across Dimensions Within CMOs20,21

Prescriptiveness Components

Group A: Tight

Instruction Group B: Tight

Group C: Tight Evaluation

and Compensation Group D: Loose CMO Mean Comparison Mean

Instructional vector (educational approach, professional development, and formative assessments)

3rd (- 0.68) 1st (0.71) 2nd (0.25) 4th (- 2.07) - 0.43 0.42

Teacher evaluation criteria

and compensation 1st (0.42) 2nd (0.32) 3rd (- 1.07) 4th (- 1.96) - 0.45 0.39

Behavior policy 2nd (- 0.44) 1st (1.21) 3rd (- 1.12) 4th (- 1.43) - 0.31 0.31

N 13 5 8 4

Source: Principal Survey.

2. Four clusters of CMOs defined by core practices

In addition to examining groupings of CMOs based on the tightness of their management approach, we also explored the extent to which CMOs fall into groups defined by their bundling of particular practices. Based on implementation of the core measures of school-level practices corresponding to six of our primary impact hypotheses, the cluster analysis suggests that CMOs can be categorized into four clusters:22

1. “Incremental Innovation” Cluster, the largest cluster, is composed of CMOs that

deviate the least from the district means across the six core practices, on average.23

20 _{The values for each of these composite measures are standardized across CMOs and district comparison groups}

such that the mean is 0 and the standard deviation is 1.

This large group appears to include two subgroups that, while relatively similar on most

21 _{The outlier CMO not included here has the following mean values: -0.98 on the instructional vector, -4.66 on}

teacher evaluation and compensation, and 1.87 on behavior policy.

22 _{The tree diagram depicted the hierarchical clustering procedure indicated that the number of clusters could range}

from three to five. An examination of the means by cluster and the CMO values within each cluster indicated that a specification of four clusters was the most meaningful, with a combination of separation between clusters and consistency within clusters.

23 _{That is, this cluster includes the fewest dimensions that deviate from the district mean by more than 1 standard}

III. CMO Practices and School Outcomes

dimensions measured, diverge in the extent to which they centralize their educational approach.

2. “Data Driven” Cluster, the next largest cluster, is composed of CMOs that are most

likely to implement performance-based compensation and evaluation and use formative assessment data most frequently. They also emphasize teacher coaching and tend to centralize instruction materials and methods.

3. “Time on Task” Cluster is composed of CMOs that tend to emphasize comprehensive

behavior policies at the school level and require the most instructional hours for their students. In addition to ranking first in those dimensions, Table III.5 shows that they also tend to coach and monitor teachers most frequently.

4. “Alternative Approach” Cluster is composed of CMOs that tend not to consistently

emphasize the key practices measured by the principal survey, ranking last among the clusters on these dimensions; these CMOs may focus on other strategies we have not captured.

The specific practices differentiating these clusters are shown in Table III.5. As in Table III.4, each row indicates one of the core practice measures, while the mean values and corresponding rankings across these dimensions are reported by the group in each column.

Table III.5. Core CMO Practices: Rankings and Means by Cluster

Cluster 1 Incremental

Innovation Data Driven Cluster 2 Cluster 3Time on Task

Cluster 4 Alternative

Approach Mean CMO Comparison Mean

Consistent behavior policy 2nd (0.35) 3rd (0.16) 1st (1.26) 4th (- 2.02) 0.30 - 0.30 Frequent use of formative assessments 3rd (- 0.03) 1st (0.68) 2nd (0.11) 4th (- 2.85) 0.03 - 0.03 Consistent educational approach 3rd (- 0.47) 1st (0.64) 4th (- 1.36) 2nd (- 0.45) - 0.31 0.30 Frequency of teacher coaching/mentoring 3rd (- 0.05) 2nd (0.91) 1st (1.14) 4th (- 1.55) 0.30 - 0.30 Performance- based compensation 3rd (0.42) 1st (1.19) 2nd (0.61) 4th (- 0.32) 0.60 - 0.62 Yearly instructional hours 3rd (- 0.16) 2nd (0.76) 1st (2.28) 4th (- 0.45) 0.41 - 0.42 Number of CMOs in cluster 19 9 5 2

Source: Principal Survey.

Although Table III.5 shows that patterns in the implementation of key practices help to define the clusters of CMOs on average, scatter plots illustrate that individual CMOs within each cluster are

indeed clearly differentiated on a few particular dimensions. In other words, these scatter plots offer evidence that, on the key dimensions, the means used to define the clusters are not masking substantially different individual CMO values within those clusters. Figures III.12, III.13, and III.14 show that clusters of CMOs are most differentiated in their emphasis on performance-based compensation and formative assessment use, behavior policy and instructional time, and teacher coaching and monitoring.

CMOs within the Data Driven cluster are consistent in their focus on formative assessment data and performance-based compensation. Figure III.12 plots CMO values on our composite measure of the use of formative assessment data versus values on our composite measure of the emphasis on performance-based compensation and evaluation; values farther to the right on the x- axis indicate greater frequency in reviewing formative assessment data, while values higher on the y- axis indicate a stronger emphasis on performance-based compensation and evaluation. The points shown as red squares depict the consistency of the Data Driven CMOs in emphasizing both practices. We can see CMOs from other clusters drifting to the upper right-hand corner of the plot, showing that an emphasis on these more data-focused practices is not entirely absent among other CMOs. However, the CMOs in the Data Driven cluster are most alike in their bundling of the dimensions shown.

Figure III.12. Data Driven CMOs Emphasize Frequent Use of Formative Assessment Data and Performance- Based Compensation

Source: Principal Survey.

CMOs in the Time on Task cluster are all consistently differentiated in their emphasis on comprehensive school-wide behavior policies and longer instruction time. Figure III.13 plots CMO values on our composite measure of a comprehensive school-level behavior policy versus values on instructional time. The Time on Task CMOs (identified by the green triangles) are concentrated in the upper right-hand quadrant, while CMOs in other clusters tend to fall below and to left.

As noted earlier, both the Data Driven and Time on Task CMOs emphasize coaching. Figure III.14 plots CMO values on our composite measure of a comprehensive school-level behavior policy versus values on our composite measure of intensity of teacher coaching and monitoring. The red squares and green triangles identifying the Data Driven and Time on Task CMOs are arrayed towards the top because of the emphasis CMOs in these clusters place on teacher coaching. The Time on Task CMOs are concentrated to the right because of their emphasis on behavior policy. - 1.5 - 1 - 0.5 0 0.5 1 1.5 2 2.5 - 5 - 4 - 3 - 2 - 1 0 1 2 Pe rf o rm an ce - B ase d C o m p en sa ti o n

Frequent Use of Formative Assessments

III. CMO Practices and School Outcomes

Figure III.13. Time on Task CMOs Maximize Instructional Time and Emphasize Comprehensive Behavior Policies

Source: Principal Survey.

Figure III.14. Both Data Driven and Time on Task CMOs Engage in More Frequent Teacher Coaching and Monitoring

Source: Principal Survey.

Although the CMOs within the Data Driven and Time on Task clusters are very consistent in their emphasis on the dimensions that typify their clusters, there is substantially more variation within our largest cluster—the Incremental Innovation cluster. Our largest group of CMOs is least distinguishable on any practices; these CMOs deviate the least from districts on these dimensions, on average. That said, the plotting of specific data points for each CMO in the group indicates substantial variation within this cluster on one particular practice: the extent to which schools rely on CMOs to centralize instruction and curriculum. In Figure III.15, the subgroup designated as Incremental Innovation - B consists of CMOs that tend toward decentralization or more school- level autonomy in educational approach, in contrast to the CMOs in the Incremental Innovation - A subgroup, which take a more centralized CMO-level approach.

- 3 - 2 - 1 0 1 2 3 4 - 4 - 3 - 2 - 1 0 1 2 3 In st ru ct io n al T im e

Comprehensive Behavior Policy

Incremental Innovation Data Driven Time on Task Alternative Approach

- 2 - 1.5 - 1 - 0.5 0 0.5 1 1.5 2 - 4 - 3 - 2 - 1 0 1 2 3 Fr eq ue nc y o f T ea che r C o ac hi ng

Comprehensive Behavior Policy

Figure III.15. Variation Within Incremental Innovation Cluster on Educational Approach

Source: Principal Survey.

A central question for this study is whether and how CMO practices contribute to greater student achievement. Chapter V examines the relationship between CMO practices and impacts on student achievement. The mechanism through which practices might affect students could be either direct (for example, extending the time allocated for academic instruction) or indirect (for example, coaching teachers on ways to collaborate with one another or make use of student assessments). In the next section, we examine two intermediate school level outcomes—instructional coherence and organizational health—that could mediate the indirect relationships between CMO practices and CMO impacts.