Relación Médico Paciente:

In estimating the effect of EMA, the problem is that the pilot and control areas may be quite different in demographic composition in ways that may be relevant for participation

decisions. If the pilot areas have individuals whose characteristics imply lower participation than in the control areas then, unless this is effectively taken into account, the impact of EMA will tend to be under-estimated. The approach for estimating the impact of EMA addresses this problem directly by using the latest matching techniques. These are an improvement on straightforward regression techniques because they ensure that only the behaviour of

individuals with characteristics similar enough to one another are compared. Regression estimates generated without careful matching might be seriously biased if the range of important characteristics vary significantly between control and pilot areas (see Heckman, Ichimura and Todd (1997)). However, matching is combined with regression techniques to carry out policy simulations (see Section 4.3 below). Next, robustness checks are undertaken using difference-of-difference estimation techniques to take account of possible unobserved area effects (under certain assumptions). For example, the labour market opportunities available to young persons could be different in the pilot and control areas. These difference- of-difference estimators are discussed in Section 4.4.

Matching is based on the assumption that all differences relevant to school participation between those in a treatment (pilot) area and those in a control area can be accounted for by controlling for observable characteristics in the survey data. The participation rate of individuals in a control area, with the same set of characteristics as those in the pilot area,

been subjected to the policy. In other words, so long as only similar individuals are compared, the control areas provide the counterfactual participation rate for the pilots. The survey data contain a wealth of background characteristics and, ideally, matching is needed on all of these characteristics. The more characteristics controlled for, the harder it becomes to find individuals in pilots and controls who are identical in all their characteristics in order to match them to one another. Following a theorem by Rosenbaum and Rubin (1985), individuals do not need to be matched to others who are identical to them in all their characteristics. Instead, a weighted index of each individual’s characteristics can be

constructed, and individuals can be matched according to their score on this index (referred to as the “propensity score index”). This allows different characteristics to be traded off against one another according to their importance to find a ‘best match’ amongst the controls.

Matching is designed to provide counterfactual outcomes based on the assumption that all differences in school participation between pilots and controls can be explained by

differences in observed characteristics only – hence the detailed and elaborate survey design including numerous characteristics.

Matching solves two problems:

1. When comparing the average outcome in the population that has been subject to the policy with the population that has not, the observations in each population receive the same weights; hence if the group not subject to the policy has a different composition from the group that is, matching reweights the samples to solve this problem.

2. The group subject to the policy may contain individuals who have no obvious

comparison group; for example, the pilot area may contain individuals from a very poor background while the control area may have none of these individuals. Matching solves this problem as well, as observations for which no suitable match can be found are dropped to ensure that the comparisons between pilot and control areas take place over a range of characteristics where suitable comparisons do exist.

It has been shown in practice that reweighting and making sure that comparisons take place over a suitable range is crucial for removing biases in evaluations. (Heckman et al., 1998). The procedure used is as follows:

1. A weighted index of characteristics, or “propensity score index” is calculated for each individual using a statistical regression technique;

2. For each individual in a pilot area an individual is located in a control area with the closest propensity score. This is the matched individual;

3. All individuals for whom a satisfactory match has not been found are deleted; and 4. The impact of EMA on individuals in pilot areas is then the difference between the

average participation rate in the pilot area and that for their matches in the controls. The average is taken over individuals for whom satisfactory matches have been found.1 The matching process needs to take place for each sub-sample of individuals of interest, so that the correct counterfactual comparison can be made. For the purposes of most of the work, it was decided to match our pilot and control samples by:

• Eligibility (those estimated to be eligible for EMA and those who are not); • Gender; and,

• Urban and rural status.

This involves dividing the sample into eight groups (eligible rural men, eligible rural women, eligible urban men, eligible urban women, ineligible rural men, ineligible rural women, ineligible urban men, ineligible urban women). For each of these eight groups, the index is estimated and each individual in the pilot area is then matched with the closest match from amongst individuals in the control area who are in the same group. An individual in a control area can be used as a match for more than one individual in a pilot area. When estimating more aggregated effects, the matching can take place at a more aggregated level.

The matching procedure which has been used means that calculating the standard errors associated with the different estimated EMA effects is very complicated analytically.2 Instead, numerical bootstrapping methods3 have been used, which allow corrected standard errors to be derived. A large number of random draws are taken from the sample, and the

1 _{The sensitivity of the results to different degrees of ‘closeness’ in matching is examined in Chapter 5.} 2

This is because the propensity score index on which individuals have been matched have been calculated using a regression approach.

3 _{Bootstrapping involves taking a random draw with replacement from the sample and undertaking the whole}

matching procedure on this sample and obtaining estimated EMA effects from each of these random draws. The bootstrapped standard error is simply the standard deviation of the mean of all these estimated effects. All

variation in the EMA effect estimated from each of these draws is used to derive corrected standard errors.