La Administración de Recursos Humanos ARH

This section gives technical details and reviews all research on the use of the RD design to estimate school effects. The RD design is not formally specified here but the specific model used is included in the Appendix D, as referred to in the results chapter.

Introduction to the Regression Discontinuity Design

The use of the RD design in social research dates back to the mid twentieth century (Shadish et al., 2002) and it has become increasingly used in educational research (e.g. Allen, 2012, Vardardottir, 2013). The use of the RD specifically for the estimation of school effects is far less common. While there are early examples (Cahan and Davis, 1987, Cahan and Cohen, 1989) of RD-based school effects estimation, the practice has only recently come to more general awareness amongst educational effectiveness researchers following research by Luyten (2006) which illustrated and assessed RD-based school effectiveness estimation and subsequent work which extended, tested and applied the method (Luyten et al., 2008, Kyriakides and Luyten, 2009, Luyten et al., 2009). Building on these promising results, researchers are beginning to make use of RD in educational effectiveness studies (Heck and Mahoe, 2010) and it is being recognised as a ‘fruitful’ methodological development in school effectiveness measurement (Reynolds et al., 2014, p.204).

RD-based measures estimate treatment effects by considering the outcomes either side of a known cut-off point for the treatment in question. A sudden break in an otherwise continuous regression line yields strong evidence regarding a programme’s effectiveness and the magnitude of the discontinuity can be used as an estimate of the programme’s effect (Trochim, 1984, Shadish et al., 2002, Bloom, 2009). This design can be applied to the estimation of school effectiveness: Many school systems admit young children to the first year of schooling on the basis of their age relative to a given cut-off date. In England, those born on the 31st August will have received a whole year extra of schooling than pupils of almost the same age born a day later on the 1st September. Within cohorts, age has a clear, positive association with academic performance and there is a strong tendency for older pupils within a particular school year to out-perform younger members of the same year group (Crawford et al., 2010). These organisational features raise an opportunity for school effectiveness measurement as, using RD, this school entry cut-off can be used to separate the effects of age and schooling and, thereby, estimate the (absolute) effectiveness of schools in improving the measured outcome (Luyten, 2006, Luyten et al., 2009).

Threats to Validity when using a RDD

Basic RD designs only need a measure of age and test scores for two consecutive year groups. Assuming a valid measure of the outcome is obtained, the key threat to validity of a RD design is non-adherence to the cut-off (Shadish et al., 2002) (i.e. pupils in a different year to that predicted by their chronological age). It is common practice in some school systems to ‘hold-back’ lower-attaining pupils by a year or to ‘promote’ higher attaining pupils to a higher year. The extent of these practices differs substantially by country (Luyten and Veldkamp, 2011). Less than 5% non-adherence is often considered a level which will give reliable estimates (Trochim, 1984). A study by Cliffordson (2010, p.50) found the effect of a non-adherence rate of 3.5% on the estimates was ‘generally relatively small’. English rates of non-adherence are generally found to be relatively low at around 1% to 2% (Luyten et al., 2008, Luyten and Veldkamp, 2011, Luyten et al., 2009).

The RD design uses the lower of two consecutive cohorts in a school as the baseline against which the absolute effect of schooling can be estimated. This raises a second problem: Cohort characteristics in a school fluctuate from year-to-year and this may lead to

unreliability in estimates of the effect of an additional year of schooling, a problem also faced by VA models (see Teddlie and Reynolds, 2000, p.72).

One final difficulty is a relative age effect within a school year. It may be the case that there are relative age effects, where being the oldest or youngest in a year group has an influence over and above this general function describing the link between performance and age. Previous research in this area, however, has concluded that the absolute age effect is approximately linear and that the pupil’s age when taking the test rather than a relative age effect is the overriding factor explaining the link between age and examination performance within a given cohort (Crawford et al., 2010, Crawford et al., 2013, Cliffordson, 2010).

Practical Use of Regression Discontinuity Designs

Use of the RD design to estimate school effectiveness is currently quite rare. Nevertheless, the existing evidence gives a positive picture of the design and its potential as well as identifying issues which must be considered. Early pieces of research showed clear absolute effects of an extra year of schooling using a RD design (Cahan and Davis, 1987, Cahan and Cohen, 1989). More recently, Luyten (2006) applied RD to the Third International Mathematics and Science Study (TIMSS), for eight countries whose adherence to the age- grade cut-off date was high and calculated estimates of grade effects (of the 4th grade). The results were in line with previous studies, showing clear grade effects, relatively high grade- age effect ratios and sizable differences between the performances of different schools. The possibility of including interaction effects in the model and thereby analysing whether other factors are associated with the ‘added year effect’ was also demonstrated.

Luyten followed this study with several other studies over the coming years exploring the possibilities of the regression discontinuity design. As well as yielding research findings in their own right, these studies demonstrated, tested and developed the possibilities of the design. One of these studies made use of PISA 2000 data and found only a small effect on reading performance in year 10-11 English secondary education and no grade effect for reading engagement or reading activities (Luyten et al., 2008). This grade effect on reading performance was not found to vary significantly between schools. In a recent follow-up study to Luyten et al. (2008), Benton (2014) points out various difficulties in the use of the OECD PISA data and found that the small effect on reading performance disappears when

these difficulties are taken into account. Benton (2014, p.10) puts this non-effect down to the lack of alignment between the PISA tests and the English secondary school curriculum.

Another example is Luyten and Veldkamp (2011) who apply RD to estimate the effect of 1 year of schooling on attitudes and achievement in mathematics and science using TIMSS-95 data, a large cross-national survey. The method is extended to include a ‘correction factor that expresses the effect of the unmeasured variables determining assignment to grades’ (Luyten and Veldkamp, 2011, p.267). They found the added-year effect of schooling for mathematics and sciences to be positive with some variation across countries. With regard to mathematics attitudes, the added-year effect was found to be negative in all cases but quite small, with RD (with the correction factor) typically explaining less than 1% of the variance. The grade effect on attitudes to science was found to be negligible, with contradictory signs and little variance explained.

Other recent studies have demonstrated the possibility of extending RD to encompass multiple-cut off points (i.e. a series of added-year effects across a number of consecutive school years) (Kyriakides and Luyten, 2009). This was achieved by Kyriakides and Luyten (2009) whose results, using both curricular and non-curricular outcomes for 577 students in 6 schools across 6 grades of secondary education, provide further evidence for the value of the RD design for the estimation of school effects. With a sample containing only 6 schools, significant differences between schools were not found in the schools’ relative effects (Kyriakides and Luyten, 2009). Also, 52 pupils of the sample of 629 (8%) were dropped from the analysis due to being allocated to year groups without strict adherence to the cut- off (i.e. were retained or promoted to another year group). Despite these difficulties, Kyriakides and Luyten (2009) provide another clear example of the successful use of the RD design and, moreover, demonstrate that it can be extended to model performance across numerous consecutive year groups rather than just two.

Of particular relevance to this study is Luyten et al. (2009), the only study known to this author which has, as is the intention here, compared cross-sectional and longitudinal estimates of school effectiveness. Luyten et al. (2009) drew on data from the baseline assessment used within ‘Performance Indicators in Primary Schools’ (PIPS) project (see

http://www.cem.org/primary, (Tymms and Albone, 2002, Tymms, 1999)), estimating the added-year and school effects for 4- and 5-year-old pupils. The PIPS data used contained ‘less than 1.5% of the pupils were in the “wrong” grade given their date of birth’ (Luyten et

al., 2009, p.146) and is therefore excellent for the calculation of RD-based estimates. Luyten et al. (2009) applied the RD design to both cross-sectional data and longitudinal data to test whether the estimates are consistent (nb. in Chapter 6, Section 6.2, these are compared alongside the VA design and are referred to as RD and longitudinal RD (LRD) measures, respectively).

Luyten et al.’s findings indicate that the overall effect (for all schools) of an additional year of schooling (for all schools) is ‘very similar’ in both the cross-sectional and longitudinal dataset for all three outcome areas and this is the case across models accounting for linear and quadratic effects of age for which there were ‘hardly any difference’ between the cross-sectional and longitudinal data (Luyten et al., 2009, pp.152, 156). In terms differences in the grade effect for individual schools, variance in the cross-sectional (RD) data was consistently higher than estimates based on longitudinal data. Correlations of the school effectiveness estimates between cross-sectional RD and longitudinal RD estimates for individual school effects were .78, .71 and .52 for reading, mathematics and phonics respectively. The latter appearing to exhibit ceiling effects in the assessment across the two years. These results suggest that school-level estimates produced by each method are fairly consistent yet have some level of disagreement. As Luyten et al. (2009) compared the estimates for the effect of only one year group for 18 schools, there is great value in replicating these results in a larger dataset including more schools and a greater range of ages.

4.4.4 Comparing Regression Discontinuity and Value-Added Designs