Determinació n de reducció n por placas

realize their potential limitations. Foremost among these is the fact that since the assessment results are fictional, teachers are unable to bring contextual knowledge to bear on interpreting results for individual students. For example, with a hypothetical set of results, a teacher cannot attribute low performance to potentially contributory factors such as the student’s language status, health status, or other disciplinary or familial problems that occur in real life. For this reason, we believe that the scenarios are best used in conjunction with a semi-structured interview with the teacher about her own assessment results. This method is described in more detail in the following section.

Winter interviews. The winter teacher interviews consisted of questions focused

on planning for and teaching during the allotted re-teaching days. Many of these questions attempted to link teacher behavior observed during our classroom visits with teachers’ intentions and with teacher use of assessment information. We also asked about professional development opportunities available to teachers since the first round of interviews and about other potential supports for interim assessment use. During this round we were particularly interested in the technological features of Philadelphia’s data reporting system that teachers use. Finally, we asked participating teachers if the term “formative assessment” was one that they were familiar with.

As part of this interview, we asked teachers to bring copies of their most recent interim assessment results with them. We asked both about class-wide patterns of performance as well as about mathematical concepts that seem to present difficulty for students. These questions were designed to closely mirror the previous questions on the Data Analysis Scenario. In this way, we hoped to get a more complete picture of

teachers’ individual capacity to make sense of interim assessment results. During these interviews, we also noted that some teachers had taken extra steps to organize their data beyond the ways in which their respective districts present this information. For example, one Cumberland teacher used the information given by the district

spreadsheet to write the names of the students who got each item incorrect alongside these items on the teacher’s copy of the interim assessment. As we analyzed our own data, we made note of ways in which teachers modified the presentation of assessment results to aid interpretation.

Spring interviews. The spring teacher interviews gave us an opportunity to

confirm trends in teacher formative assessment use that we had begun to identify, specifically, to further explore teacher use of interim assessment results to understand student thinking and to help identify their own professional development needs. We also used this final round of interviews to ask teachers about the role of classroom

assessments in light of the annual state assessment (the PSSA) that had just been administered.

In order to gain a broader and deeper understanding of teachers’ use of interim assessment results, we linked several questions to two types of artifacts in our spring interviews: an item from the most recent interim assessment and (in Philadelphia) the Benchmark Data Analysis Protocol (BDAP), a two-page, district-created analysis and reflection worksheet. We chose one item from each of the most recent four interim assessments by selecting those that we felt offered teachers the most opportunity to learn about student understanding of mathematical concepts. To our surprise, it was difficult to identify items for which the distractors offered meaningful information about student learning or open-ended items that targeted mathematical understanding. Out of 20 items on each assessment, only 2 to 3 were identified as potentially informing knowledge about student understanding relative to a learning goal (as opposed to merely indicating, for example, that a student could or could not perform a procedure). We then chose one item from these 2 to 3 based on the relative curricular importance of the mathematical content contained therein (e.g., operations were given precedence over measurement) and on the perceived difficulty of the item. Much as we had done in

the fall and winter, we asked teachers to describe what students who got this item incorrect might have been thinking, what steps the teachers would take to confirm or disconfirm this hypothesis, and how they might address student misunderstanding. Because much has been written about the greater potential for constructed-response and open-ended items to reveal student understanding relative to multiple-choice items, and because the Cumberland interim assessments contain constructed-response and open-ended items, we also asked Cumberland teachers to bring examples of (incorrect) student work to the interview. We then asked these teachers a similar set of questions aimed at understanding what knowledge they gained from such items and their planned instructions responses (if any) to such misconceptions.

In Philadelphia, we asked several questions about the BDAP and reflection worksheet. While completing the reflection component of the protocol was officially voluntary, in the Spring 2006 background interviews, all of our participating principals reported that they expected their teachers to complete these forms and hand them in to them. Our particular interest in the BDAP was whether or not teachers used it to report their own professional development needs and whether or not school leaders used the resulting information to provide requested assistance.

School and district leader interviews. In addition to observing classroom instruction and interviewing teachers, we also conducted background interviews with district and school leaders in Spring 2006 and Spring 2007. District respondents included directors of assessment, accountability and curriculum and assessment, and district-level instructional coaches (n=10). School leaders included principals and school- based instructional coaches (n=25). We used these interviews to understand the context for the use of interim assessments, and to identify district- and school-level expectations for interim assessment use and potential supports offered to teachers (e.g., additional days for re-teaching; release time for professional development; data management

systems). We were also interested in how school and district leaders used the data reported from the interim assessments in their own roles. This information enabled us to make connections (and identify mismatches) between school-and district-level

expectations, support and data uses, and classroom expectations and uses of interim assessment results.

Observation of district and school meetings. In order to gain a more complete understanding of assessment creation and data use at the school- and district-level, we attended principal meetings in Philadelphia at which several types of “performance indicators” were discussed and test-construction meetings in Cumberland. We also observed several professional development workshops for both teachers and principals. Finally, because grade-level collaboration was mentioned as a possible support for teacher interim assessment use, we attended a handful of grade-group meetings in three of the Philadelphia schools. In each case, we took field notes and used this information to triangulate findings from the teacher and principal interviews and to help contextualize our findings from teacher observations and interviews.

Collection of relevant artifacts. We collected copies of all 3rd- and 5th-grade interim assessments in mathematics given in the 2006-07 school year. We also

purchased the 3rd- and 5th-grade EM program in order to better understand the learning goals that were to be achieved during this study. We acquired copies of both districts’ pacing guides and of any additional district-wide assessments that were made available to us. In many cases, we collected examples of teacher classroom assessments and, in some cases, teachers offered blinded examples of student work on the interim

assessments. As mentioned above, a few teachers had constructed their own data organization templates, and, when possible, we collected copies of these as well.

Teacher survey: Content Knowledge for Teaching-Math (CKT-M). Following our final teacher interview, we distributed a survey designed to measure our participating

teachers’ mathematical knowledge for teaching. This survey was composed of nine multiple-choice items from the CKT-M instrument, focusing on K-8 numbers and operations. The CKT-M was developed by researchers at the University of Michigan to measure “the knowledge teachers use in classrooms, rather than general mathematical knowledge [emphasis in original]” (Hill, Rowan, & Ball, 2005, p. 387; see also Hill, Shilling, & Ball, 2004). We chose to administer the CKT-M to our sample of teachers so that we could investigate the relationship between this specific type of mathematical content knowledge and teachers’ use of information from interim assessments in mathematics.

The CKT-M creators chose nine items to maximize reliability while lessening the time burden on teachers to complete the survey. Information gained from these items was intended to be sufficient to categorize participating teachers into three groups: those with high-, average-, and low-mathematical knowledge for teaching, relative to the other teachers in our sample. Teachers received a $25.00 gift cheque upon completing the CKT-M, and response rates were high. The response rate for Philadelphia teachers was over 90%, and for Cumberland teachers was 64%.