La estrategia publicitaria

3.2.4.1 Evidence sources

Monitoring, reviewing and evaluating the impact of interventions to support the most academically able disadvantaged pupils was normal practice in the participating schools that identified such a group. (The one exception was the school where this focus was very new and where the number of such pupils was very small.)

There were four main ways of doing this (evidence sources).

45

Figure 18 How these schools monitored, reviewed and evaluated the impact of their interventions for this group

Evidence source Composite details and examples Data on academic

progress

Internal school assessment results: monitored regularly against ‘flight path’ or ‘tracking’ data – several schools mentioned setting targets above those expected by prior attainment tracking

models

External exam results: GCSEs; ultimately A-levels

Pupils’ views Pupils’ complete evaluation sheets after an intervention; speak with students one to one or in small groups to seek their views of impact; annual survey of Year 11 pupils to gain their

retrospective views on what helped them

Views of key staff Views of form tutors, of Heads of Year, of subject teachers – based on their knowledge of individuals and/or this group. Data/perceptions re improved: attendance, behaviour (e.g. measured through reduced sanctions or increased house points or equivalent), engagement, well-being, involvement in extra- curricular activities

Progression data Numbers applying to university; numbers accepted by

universities; numbers going on to Higher Level Apprenticeships (degree equivalent)

Source: Phase 2 and Phase 3 interviews (examples drawn from different schools) One additional way was mentioned by only two of the participating schools: this was to seek the views of the school’s partners in an intervention: the parents of pupils involved, and any employers or other external organisations involved.

46

3.2.4.2 Examples of evidence of impact on attainment

The types of evidence of impact on attainment were, of course, in line with what evaluation data were collected.

Figure 19 Examples of evidence of impact on attainment of most able disadvantaged pupil group Type of data Examples of evidence of impact on attainment for

most able disadvantaged pupils Reading ages “Huge improvement” (School 22)

Attainment gap The gap has “narrowed” (School 11); The gap is “closing’ (School 21); “close [...] and, at the moment, surpass the gap: our disadvantaged groups are performing better than the rest of our cohort [... due to] very tailored support” (School 17).

Progress 8 score 2016 Progress 8 for high prior attainment (HPA) Pupil Premium pupils was 0.19: in 2017, it was 0.5 i.e. “that was a serious impact on that group” (School 22).

2016-17 Progress 8 score for HPA Pupil Premium group showed a gender gap: HPA Pupil Premium boys had fallen way below the HPA Pupil Premium girls: in Y11, 10 males – Progress 8 prediction was -0.64: 7 females it was +1.26. (School 27).

2016-17 Progress 8 score of 0.33. (School 33) Percentage of total Pupil

Premium cohort going on

to any university 45% (School 7) Qualitative data from

school staff

Year 7 most academically able disadvantaged students took part in a national debating completion. Head of Year reported that their confidence had grown as a result. (School 1)

47

3.2.4.3 Some issues raised relating to evaluating impact

Gender: Several schools mentioned the national issue around the attainment gap for White working class boys. One school provided some data to evidence this gap in their own results. In this school, the 2016-17 Progress 8 score for Pupil Premium group was comparable with the non-Pupil Premium group (i.e. overall Progress 8 was 0.52 and for Pupil Premium only (N=39) it was 0.51. The interviewee noted a gender gap within that: 22 boys: 0.36; 17 girls: 0.7.

“The current issue is that the HPA [high prior attaining] PP [Pupil Premium] boys have fallen way below the HPA PP girls: in Y11, 10 males – Progress 8 prediction is -0.64: 7 females it is +1.26.” (School 27 Interviewee 21)

The perennial issue of attribution: Several of the school staff interviewed raised the issue of how difficult, if not impossible, it is to directly attribute measurable gains in academic progress to specific activities or to a suite of specific activities aimed at the most

academically able disadvantaged pupils:

“[...] like anything in school, cause and effect, that direct correlation, is very hard to say but I would say that, as an SLT [senior leadership team] sat there, we said ‘OK, the indications are that this is working, and it’s worth us continuing with this particular programme’.” (School 7 Interviewee 68)

The attribution issue was a key reason why many of the schools included qualitative data in their evaluation of their interventions: if the pupils involved articulated that the

intervention had made a positive difference to them and there was measurable academic progress, school staff felt more confident about there being a link between the two. Qualitative data was also gathered in relation to particular cultural extension or personal development activities where ‘hard’ data is lacking. For example, in relation to evaluating the impact of pupils attending a national competition, one school lead noted:

“[...] there’s not so much hard data but what we will do is we always look at

students and we ask students to tell us. So we’ll ask them for a simple, ‘What went well? Even better if ...? [EBI]’. We’ll ask them to rank what they thought of the experience and how it’s impacted upon them. We’ll assess it that way.” (School 1 Interviewee 7)

Pupil voice was also viewed as of value, in and of itself, by a number of the participating schools. For example, one interviewee put it like this:

“I do a pupil voice questionnaire after each intervention has taken places, and I do it at the start and end of each academic year to ask them their thoughts, what has gone well, what hasn’t gone so well, what are they taking on board, what things could they improve on (both the school and what they could learn more about).

48

For example our post-16 student understanding more about the UCAS process. Lower down her school it gives them the opportunity to raise their own aspirations and have the conversations that potentially they won’t have at home. So they are invaluable from our point of view.” (School 28 Interviewee 8).

Improved practice benefiting all – and raising the bar around closing the gap: Another issue raised around evidence of progress and impact on the most academically able disadvantaged pupils was that

“I still would say that our data suggests that there is a gap, though it’s a narrowed gap and the rising tide of our performance at school has raised all boats including the disadvantaged. Though I would still say, clearly, there is still something in disadvantage that we haven’t wholly managed to pin down. So it’s an ongoing, continual battle and focus of everything that we do, I would suggest, even still.” (School 11 Interviewee 20)