Fundamentos de teledetección aplicados 47

There are several limitations of using school-level models. One of these is that they preclude examination of pupil-level differences or differences within school. As well as limiting the possibilities for research, use of school-level results potentially masks substantial differences in performance in relation to different pupil groups. With the creation of the National Pupil Database and widespread use of performance data systems within schools in England (Kelly et al., 2010), pupil-level data are now commonly available for analysis and can easily be aggregated to school-level when this is required. As a result, the debate about the seriousness of the limitations in school-level models is no longer current and pupil-level data are used. Further details regarding the limitations of school-level models are given in Appendix A.

There are several statistical approaches to estimating relationships between non- school factors and measured school outcomes. This section describes a) the common regression based ‘ordinary least squares (OLS)’ approach and b) the technique used for the forthcoming English Progress 8 measure (Burgess and Thomson, 2013a). Further approaches are examined in the technical document accompanying the report for the Progress 8 measure (Burgess and Thomson, 2013b).

Model Specification 1- OLS

One specification of a pupil level model uses ordinary least squares (OLS) multiple regression. This applies the school-level approach described above, to pupil-level data, where each data point represents a pupil rather than a school average figure. To get school- level value-added scores, the school mean of these pupil-level residuals can be calculated. The underlying equations for this are given in Appendix A2.

As noted in the previous section, models are typically extended to include a greater range of non-school factors in addition to prior attainment to further isolate the school effect from other confounding factors (see Appendix A2 for technical details). There are many non-school factors which have been consistently associated with school performance over several decades of educational effectiveness research (Teddlie and Reynolds, 2000). Prior attainment is the most important of these: As well as reflecting a direct effect of previous performance supporting future learning, prior attainment acts as a ‘black box’ which reflects

a large number of unobserved factors which have brought about the differences in previous performances. There are several other factors which also tend to be associated with rates of progress over time, over and above those feeding into prior attainment (Teddlie and Reynolds, 2000). For example, the official 2007 contextualised value-added (CVA) measure accounted for associations between performance and deprivation, local area deprivation, in care status, special educational needs status, pupil mobility, gender, age within year, English language status, ethnic group and school average prior attainment (Evans, 2008).

Adding extra contextual variables explains more of the overall differences between pupils’ performances and so reduces the size of the residual variation which is used as evidence of value-added. Kelly and Downey (2011b, p.64) estimated that a KS2-KS4 value- added (VA) model controlling for prior attainment accounts for 49% of the pupil performance variance while a CVA model accounts for about 57%. Accounting for a greater range of contextual variables makes greater demands of the available data (Kelly and Downey, 2010) and it is often difficult to control for non-school factors without also attenuating the value-added scores (Visscher, 2001) (see Chapter 4).

Model Specification 2 – Progress 8

The forthcoming English ‘Progress 8 measure’ (due 2016) uses a different estimation approach to that used in the regression-based models above. The approach described in the last section used an equation to fit a regression line to estimate the relationship between prior and final attainment. In contrast, the Progress 8 measure calculates the mean pupil KS4 point score for every possible pupil KS2 score for all pupils in the national cohort. The underlying measures of overall performance used for this are the KS2 average point score and the KS4 Attainment 8 measure (not discussed further here). Mapping of KS2 scores to KS4 scores in this way is analogous to fitting a regression line, above, but will produce an irregular (as opposed to linear or curvilinear) expectation line. In practice, the result closely resembles the estimate that would be produced using a non-linear regression line (Burgess and Thomson, 2013b). See Appendix A2 for a technical note on how to best map the KS2 scores to the KS4 scores.

The big advantage of this approach is that it is very simple and the correspondence between KS2 and expected KS4 scores can be provided in simple table or graph. Moreover, the resulting estimates and the proportion of the variance explained using the method is

almost identical to approaches based on far more complex techniques (Burgess and Thomson, 2013b). One limitation is that the model cannot be extended to include a greater range of control variables (see above) without losing this simplicity, although subsequent adjustment is possible.

Outputs

Both the OLS and Progress 8 methods can be used to produce pupil-level value-added scores from the difference between the actual performance and the expected performance generated from each fitting method. If school-level value-added is required (e.g. for the English school performance tables), the school means of pupil-level results can be calculated. It is also possible to work with pupil-level value-added scores if this is of interest. For example, the mean value of groups of pupils within the school such as ethnic groups or classes can be found or the overall distribution be viewed graphically or summarised using key statistics.

Researchers are likely to include other educational effectiveness factors to examine their relationship (causal or otherwise) with performance. The simple Progress 8 approach does not allow for this kind of research – or at least without taking subsequent steps with the results. The OLS model, however, can be extended to examine factors which are associated with school performance. Factors can be included at various levels of analysis using the teacher-, school- or regional-level means, subject to their availability and concerns of the research in question. The inclusion of variables for purposes of study will generate estimates for variable coefficients within the model. These are an important output of the model in the research context, where the aim is often to understand the association between educational factors and performance.