Discusión de resultados

 The concept of propensity score matching and its advantages

Leyland (2010) stated that “If the community is the unit of intervention then it is at the community and not the individual level that balance must be achieved”. With this in mind, to provide a measure of the counterfactual, the next stage of the analysis involved selecting appropriate control CATT areas. Areas - as opposed to individuals - were used as the unit for matching due to the nature of the ‘treatments’ of interest, i.e. regeneration initiatives that specifically targeted areas. This was carried out using the propensity score matching technique.

The propensity score matching technique aims to balance two non-equivalent groups based on observed covariates to gain a more precise estimate of the effects of a treatment on which the two groups differ (Luellen 2005). The technique was first outlined by Rosenbaum and Rubin (1983) who showed that matching on a single index that reflects the probability of participation could achieve consistent estimates of the treatment effect in the same way as matching on all covariates. This index is the propensity. Thus Rosenbaum and Rubin (1983) defined the propensity score as the conditional probability of receiving treatment given the set of confounders and the approach shares a number of assumptions with regression based approaches. The clear advantage of propensity score matching is that it replaces high-dimensional matches with single-index matches. The propensity score reduces the discrepancies observed in the characteristics of treatment and control groups, and thereby reduces the bias in estimation of the treatment effects with observational data like surveys, administrative records and census data. However, matching on propensity scores can suffer if poorly measured variables are employed to obtain the propensity score (Bryson et al. 2003). In this case, deprivation variables from the 1991 census were used to ensure the variables were of good quality.

 How the Propensity Score Matching process is conducted

According to Barth et al. (2008) conducting a process of propensity score matching involves a three- step analytic procedure. The first step involves estimating the propensity score using logistic regression. Here a logistic regression is used to identify factors ‘predicting’ exposure to the intervention. The model is used to calculate each individual’s predicted probability of, rather than actual, exposure to the intervention (Cousens et al. 2003). Thus, the predicted probability is the propensity score which in the case of this study is the probability of being in a SARP area.

The second step involves matching treated subjects to non-treated subjects on the basis of the estimated propensity scores. Barth et al. (2008) explained that after estimated propensity scores are obtained, cases are matched to create a new sample of cases that share approximately similar

63 likelihoods of being assigned to the treatment condition. Thus, individuals or areas with similar propensity scores are grouped and within each group, some individuals will actually have been exposed to the intervention and some not. Since individuals in each group had the same propensity to be exposed, the method assumes that actual exposure within these groups was random (Cousens et al. 2003). There are currently multiple forms of matching in use; however in this study the matching method used was the nearest neighbour method with caliper. This method is conducted via the following three stages:

i. Firstly the treated and non-treated subjects are ordered randomly.

ii. Secondly, the first treated subject and non-treated subject with closest propensity. Score is selected within the pre-determined common support region called a caliper.

iii. Thirdly, both subjects are then removed from consideration for matching, and the next treated subject is selected (Barth et al. 2008).

Nearest neighbour matching within a specified caliper specifies that the absolute difference in the propensity scores of matched subjects must be below some predetermined threshold called the caliper distance. Austin (2011:404) explained how this works:

“for a given treated subject, one would identify all the untreated subjects whose propensity score lay within a specified distance of that of the treated subject. From this restricted set of untreated subjects, the untreated subject whose propensity score was closest to that of the treated subject would be selected for matching to this treated subject. If no untreated subjects had propensity scores that lay within the specified caliper distance of the propensity score of the treated subject that treated subject would not be matched with any untreated subject. The unmatched treated subject would then be excluded from the resultant matched sample.”

The third step of the process according to Barth et al. (2008) is analysis of the treatment effects based on the matched sample to answer the research questions of the study. Thus at this stage bivariate or multivariate analysis is conducted to compare outcomes between the treated and comparator groups in order to assess treatment effects for the treated group.

 How the propensity matching was applied in this research

Thirty-nine variables (See Appendix 5) were extracted from the 1991 census and the PSMATCH2 function in STATA 10 was used for the propensity matching procedure (Leuven and Sianesi, 2003) in order to identify places that matched the characteristics of the regeneration areas. The areas that received SARP regeneration were significantly deprived areas that were chosen for funding following a bidding process. This made it possible to conduct this quasi-experiment as there were other similarly deprived areas in Scotland that were not allocated funding because a bid was either not submitted or bids that were submitted were considered unsuitable. It was therefore important to

64 identify comparator areas that were also significantly deprived. The 39 variables listed in Appendix 5 therefore included measures of area deprivation across 4 domains (Housing, Access, Employment and Health) and also population characteristics in order to match areas on the demographic composition of their populations, thus increasing the ability to make accurate comparisons. The deprivation measures include items used in well-known deprivation scores such as the Carstairs and Morris Scottish deprivation score (Carstairs and Morris 1991) (e.g. Overcrowding, Male unemployment, Social Class IV or V, No car). Thus, the purpose of choosing these variables was to have the areas closely matched on as many dimensions as possible.

The following provides an overview of steps taken to achieve the treatment and comparator groups:

i. First 39 1991 census variables were extracted from CASWEB, a resource based at the University of Manchester that allows the downloading of aggregate UK census statistics and digital boundary data developed by the Census Dissemination (http://casweb.mimas.ac.uk/). These variables were chosen for their association with treatment decisions and outcomes. The full list of variables can be viewed in Appendix 5.

ii. These 39 variables were aggregated into the CATTs system. iii. Logit regression was conducted to estimate propensity scores.

iv. The comparator areas were then created using the PSMATCH2 facility in STATA 10.1.

Three types of comparator areas with differing geographical characteristics were created with the aim of assessing the strengths and weaknesses of different matching criteria. All three sets were created using the ‘nearest neighbour’ matching technique with caliper. Care was taken to ensure that the comparator areas were not included in the SARP areas, and to the best of our knowledge, they were not included in any other area-based-initiatives in the past. The nearest neighbour matching technique randomly sorted the treatment and potential comparator CATTs before the first treatment CATT was chosen to find its closest control match based on the value of the difference of the logit of the propensity score of the selected treatment and the comparator under consideration (Coca Peraillon 2006). The closest comparator CATT was then selected as a match. This process (which is then repeated for all the treatment CATTs) made sure that each treated CATT found a match even if the propensity scores are not close provided there are enough comparators available. In the nearest neighbour matching a caliper was imposed whereby treatment and comparator CATTs

65 were only matched if the comparator’s propensity score is within a certain radius. There is no uniformly agreed upon definition of what constitutes a maximal acceptable distance for a caliper (Austin 2011), in this case one fourth of the standard deviation of logit of the propensity scores was used for the caliper radius, which has been suggested in Guo (2005). Consequently, use of the caliper method means that a treated CATT may not be matched to a comparator as the aim is to avoid poor matching.

The following table shows the central features of the three comparator CATTs:

Table 3-2 Characteristics of the comparator groups

Comparator Group Matching Type Conditions

1st Set One to One Nearest Neighbour With grid references (X,Y) as co- variates, which favours areas close to regeneration areas.

2nd Set One to One Nearest Neighbour Without grid references (X,Y) co- variates, geographical location is not taken into account.

3rd Set One to One Nearest Neighbour Without grid references (X,Y), excluding CATTS contiguous to regeneration CATTs

These 3 sets of comparator areas were then checked using t test to compare with the regenerated CATTs based on the 39 census variables in order to ensure that they were well balanced with the treatment areas on all variables (i.e. no significant difference on any of the 1991 census variables between treated and comparator areas). The results of this balancing can be viewed in Appendix 5.1 and show that regeneration and comparator residents were balanced on all characteristics. In order to determine which set of comparator areas should be used in the main analyses, three issues were reviewed. First, the issue of location in the same local authority area suggested by Cotterill et al.

(2008). The 1st set of comparators are based on criteria that favour areas proximate to the regeneration area (Table 3-2) and therefore most closely accord with this suggestion. However, as was seen in Figure 3-3 above, the SARP areas are mostly concentrated in the Central Belt of Scotland where the majority of the population resides. Arguable, SARP areas in other parts of Scotland (such as in Aberdeen and Dundee) have more in common with the kind of deprivation found in the Central Belt rather than with more proximate areas. It was therefore felt that location in the same local authority area was less important than other criteria. Second, the possibility of ignoring geographical

66 location was considered. The 2nd set of comparator areas generated took no account at all of location. However this was considered unsatisfactory due to the fact that there was potential for these comparator areas to be located contiguously to regeneration areas, which therefore leads to the third issue of ‘spill-over effects’. For example, Gutierrez-Romero and Noble (2009) noted that the implementation area regeneration policies at the small area level can impact on households not directly participating in the programme due to spill over effects. They argued that these effects are likely to occur when the involvement of residents in regeneration activities enhances social networks, creating links between both participants in regeneration activities and non-participants. Any comparator area affected by spill-over would confound the quasi-experimental analysis because this area will have a chance of its residents having received some benefit from the programme, thus confounding the attempt to ascertain what would have happened in the absence of the initiative. It was therefore concluded that the avoidance of such confounding would be the prime criteria for choosing comparator areas. Thus, the 3rd set of comparators excludes areas geographically contiguous to regeneration areas, which ensures they the selected comparators are unlikely to be affected by spill-over effects. This issue was not addressed in the definition of either the 1st or 2nd set of comparators. It was therefore decided that, on balance, selecting the 3rd set of comparators was a sensible approach in that they are not contiguous to regeneration areas and thus control for any potential spill-over effects from regeneration areas into comparator areas. Whilst this choice does not guarantee that comparator areas are in the same local authority, the geographical distribution of deprivation in Scotland means that most will be in the Central Belt. In addition the SARP programmes were national level policies with the same aims and objectives in each region. Thus it is the author’s contention that controlling for possible spill over effects was the most important issue in regards to selecting counterfactual areas for this analysis. Once the 3rd set of comparators was selected, the treatment and comparator areas were attached to the individual-level data from the Scottish Longitudinal Study (SLS) in preparation for the main analyses.

The following table (Table 2) shows the number of individuals resident in regeneration areas, the 3rd set of comparator areas and the rest of Scotland in 1991 and 2001:

Table 3-3 SLS Sample Members by Area

1991 2001

Variable Frequency % Frequency % Regeneration Areas 39,622 14.64 36,868 13.86 3rd set of Comparator

Areas

28,529 10.54 25,287 9.51

67