A good set of measures would gauge overall effectiveness against those objectives and enable comparison between different sets of arrangements. We take in this report three objective measures, the level of appeals, the proportions of parents gaining which of their expressed preferences and the levels of segregation. The advantages and limitations of these measures is discussed below.
Appeals as a measure of effectiveness
We can reasonably assume that the greater the number of appeals the greater the dissatisfaction of parents with the original allocation and vice versa. The number of appeals works as an overall measure of parental satisfaction and therefore of effectiveness. An effective policy would result in acceptable offers to all parents using the initial procedures without recourse to the appeal process which is costly in the time of many professionals from the local authority and schools. A system that resulted in fewer appeals would therefore be more administratively and financially efficient and effective as well as in terms of parental satisfaction. But it is important to note that such an outcome is determined not just by the system in the sense of ensuring adequate information, administration or regulation important as these are. Parental perception, expectations and consequent behaviour are what ultimately determines parental satisfaction and these are affected by the way choice of school is framed by their perception of the local schools and through public discussion and policy formation at both national and local levels.
However, the validity of appeals as a comparative measure between the arrangements in different authorities works for some but not all Local Authorities.
The level of appeals for a Local Authority’s schools tells us only about the level of satisfaction of the group of parents who made applications for those schools.
These are not necessarily the same set as those parents who are resident in the area. Indeed, in one London area where the schools are sought after, and where there are adjacent areas with less popular schools, up to half of applicants are non-resident and half of the resident parents apply to schools outside the Local Authority. In this case there can be no sound conclusions drawn from the level of appeals about the level of satisfaction of those Local Authority residents with the arrangements in their area or of any other identifiable cohort of parents, or of the effectiveness or otherwise of the admission arrangements in that Local Authority.
However, where there is a low level of cross border traffic and the set of applicant parents is largely the same as the set of resident parents, and the field within which all parents are choosing is more or less the same for all parents, then the level of appeals does offer a way of comparing the effectiveness of the arrangements in that area.
Parents' expressed preferences as a measure of effectiveness
It is commonplace in debate about admissions to take the proportion of parents allocated the school that was their top ranked expressed preference (first preference) as a sign of the effectiveness of admissions. If a parent gets the school they put down as their first preference then it is reasonable to assume they will be satisfied. It is difficult to claim that choice exists if a significant proportion of
parents do not get the school they put as their first preference. There are however some difficulties that need to be considered.
Firstly the time for which the data is collected can affect the validity of the measure. Some of our respondents pointed out that the data for March 1st (the national offer day) do not reflect the proportion of parents who eventually get their first preference following the whole admission process including appeals and therefore may not properly reflect the real effectiveness of the system. This is true, but while all authorities have data for March 1st many do not have secure data for the end of the process. It is also the case that the March 1st figure better reflects the way the initial allocation works and this is of more relevance for judging the operation of such aspects as over subscription criteria. It remains true however that in most cases there is an improvement in the percentage between March and the beginning of the school year.
Secondly, two authorities with a high proportion of selective places were worried that the figure for first preferences would not accurately reflect the level of satisfaction delivered by the system because a proportion of parents put a selective school down as first preference but, if their child doesn’t reach the right level in the test they are not then eligible for the grammar school. They argued that if, in what is misleadingly called an 'equal preference system' where the ranking of schools is used only if the child is eligible for two or more schools, the child is then offered a place at their highest preference comprehensive/secondary modern then this represents their ‘first’ choice. We felt that, since the grammar school was their first choice and the admission criteria had resulted in their not getting their first preference it was more accurate to count them as having gained their second choice.
Thirdly, although it does not invalidate the measure, it should be noted that a small proportion of parents do not put down the school they most preferred as their first choice on the Common Application Form. There may be a school within their field of choice that is their favourite school but, because they make a strategic decision, taking into account how likely they are to get in, they do not put it as their first or any preference. Because of this possibility the parent survey in both the 2001 and current study asked parents about this. In the 2001 study 8% of parents reported that there were other state schools they would have preferred for their children over the ones in which they had applied for a place, and there were regional differences, with London parents least likely to get a place in their favourite school.
A more effective system would have a lower percentage of parents who reported doing this. However the numbers are small and the reasons for not doing so can be various.
Fourthly, some non-selective authorities that have an equal preference/default ranking system returned our request for data on each of the preferences saying that because it is 'equal preference' they do not distinguish which of the schools is most or least preferred by the parent. This we believe is an incorrect interpretation and if put into practice would badly misrepresent the wishes of most parents. It is not reasonable to assume that if a parent is invited to make a list of preferences each one of the schools on that list is equally acceptable and, that if they get one of their expressed preferences, this should count as their choice having been
satisfied. It is arguable that those who get one of their first two preferences should be counted as gaining a preferred school but this becomes difficult to sustain for lower ranked schools. But, the only sure way of knowing if parents are satisfied with the school they have been offered, wherever they put it in their ranked expressed preferences, is to ask them.
We therefore sought two measures. On March 1st, the proportion of first preferences gained by applicants in each Local Authority which gives a strong indication of the satisfaction level of the cohort of applicants to that authority but where there are a great deal of cross border applications, as in London, it does not allow us to compare the effectiveness of the admission arrangements of an area in terms of the satisfaction of residents of an area. The aggregate figure for all Local Authorities gives a reasonable approximation of the overall satisfaction level.
Secondly, the responses from a nationally representative sample of parents as to how satisfied they were with the outcome and which of their preferences it was.
Segregation as a measure of effectiveness
We have seen that equity is one of the objectives of an admission system. No parent or group of parents should more easily gain access to ‘good’ and popular schools than any other parents. We have seen that there are problems in achieving this kind of equity and that socially and financially advantaged parents may well be gaining access more easily than others. Although the problem is difficult to solve it is fairly easy to measure. If all social groups had the same chances of getting into the ‘best’ and most popular schools the intakes of those schools would reflect the proportions of social groups in the Local Authority area.
There would be no segregation. The more segregation of the intakes of popular schools the less fair or equitable the system.
Measurement of segregated populations is fairly well established and, although the technicalities with regard to the segregation of school intakes have been hotly disputed, the results from different studies are consistent. We have in this study taken Allen and Vignole’s (2006) method of using two measures. The first, D (Difference), measures how many children in an area would need to move between schools to achieve a fully balanced intake. This provides a single value for each authority and allows us to explore associations between the degree of segregation in an area and the different kinds of admission arrangements. A difference measure cannot however capture whether the level of segregation in a highly segregated area is due to there being a few schools with highly advantaged intakes with the rest with more or less similar intakes (advantage skew) or, a few schools with extremely disadvantaged intakes (disadvantage skew) or, a smooth continuum from highly advantaged moving by even differences through to highly disadvantaged. The second measure S (Skew) captures this aspect.