The main steps involved in carrying out a binary propensity score matching (PSM) are as follows: (a) Obtain a dataset including information on basic characteristics and performance of programme
beneficiaries and non-beneficiaries in two time periods, i.e. prior to implementation of a given RDP and after. One possibility to pre-select individual units of programme non-beneficiaries to a dataset is to apply programme or measure eligibility criteria.
(b) Compute differences in all basic characteristics and performance of programme beneficiaries and non-beneficiaries prior to applying matching.
(c) Run a participation model (probit/logit regression). Generally, covariates entering the probit/logit function are expected to determine both programme participation and outcomes (the latter are typically measured in terms of relevant result indicators at micro level).
(d) Calculate participation probabilities for each individual unit (programme beneficiaries and non- beneficiaries) included in the dataset.
(e) Drop observations outside the region of common support (i.e. individual observations in the group of programme beneficiaries whose probability of receiving support exceeds that of any from the potential comparison group, or those from the control group with probabilities of receiving programme support below those of any members of the group of programme beneficiaries).
(f) Match observations based on participation probabilities (here various matching algorithms can be applied. Selection of appropriate matching algorithm should be a subject to statistical analysis, e.g. by i) applying % of the standardised bias reduction – after matching -; or ii) applying pseudo R2 test after matching - as a selection criterion).
(g) Calculate programme results for each pair or set of matched observations.
(h) Calculate the average of these differences for a period prior to and after programme implementation using ATT combined with DID.
Table 22 Comparison of a binary PSM with a generalised PSM approach
Binary Propensity Score Matching
Main hypotheses Data required Pros Cons
Programme selection is determined only by observable characteristics not influenced by
programme support (yet, proxy variables can be included as controls for unobservables).
The dataset contains all the variables describing major characteristics and performance of
programme beneficiaries and non-beneficiaries prior and after implementation of the programme.
Very effective tool applied in impact evaluation of various programmes.
Very effective tool for finding counterfactuals.
It controls a set of covariates that simultaneously
determine the decision to participate in a given programme.
Is a non-parametric approach, therefore very flexible (i.e. matching does not require any functional form assumptions for relationship linking the outcome variable with the covariates).
It can only be performed on observed
characteristics.
It will not generate reasonable results if other important observable characteristics which explain differences between programme beneficiaries and control group were not included in the model.
It is a data-hungry procedure.
Its conclusions hold only on the subset of matched units.
The external validity of its results decreases when the share of unmatched units increases.
Generalized Propensity Score Matching
Main hypotheses Data required Pros Cons
Programme selection is determined only by observable characteristics not influenced by
programme support (yet, proxy variables can be included as controls for unobservables).
The dataset contains all the variables describing major characteristics and performance of
programme beneficiaries prior and after
implementation of the programme, and intensity of programme support obtained during implementation of RDP
It does not require existing units that did not receive programme support;
Very effective tool applied in impact evaluation of various programmes if almost all units receive a programme support (at various intensity levels).
Very effective tool for finding counterfactuals.
It controls a set of covariates that simultaneously
determine the obtained intensity level of programme support.
It allows to assess the result/impact of a given programme for each relative level (between 0% to 100%) of obtained support
It will not generate reasonable results if other important observable characteristics which explain differences between various programme beneficiaries were not explicitly included in the model.
It requires high analytical skills.
9
TOOL TO IDENTIFY DATA SOURCES FOR EVALUATION
The table below intends to support MAs and evaluators in identifying data sources for RDP evaluation, in particular for evaluation methods applying counterfactual. Once the table is filled with all known data sources, it shows where data gaps occur and which need to be bridged, either by the MA or the evaluator.
The table is subdivided in five categories according to the potential RDP beneficiaries: 1) agricultural holdings, 2) forestry holdings, 3) food holdings, 4) villages and, 5) micro-enterprises, NGOs. The required data sets should always cover the total population; therefore include RDP beneficiaries and non-beneficiaries.
The data listed in the tables is not exhaustive and may not cover all areas needed for specific RDP evaluation. Therefore MAs are encouraged to add items or change the list according to their specific evaluation needs or methods employed.
For all the data sources, it is advisable to collect the following information in order to have complete and quality full data sets:
Description of data
Population size (number of units for which data are collected)
Owner
Web link (URL)
Measurement unit of data (e.g. EUR, tons, kg/ha)
Availability of database(e.g. on the web for free; on the web for purchase; available on request (free); available on request (purchase); not available)
Name of database
Remarks
In the tables below the data source table is filled out for the Slovak Republic, the first four columns are presented.
Agriculture
Agricultural Farms (total population – beneficiaries and non-beneficiaries )
Population size Owner Weblink (URL)
Data on obtained level of support from each RDP measure (by measure and total)
national, NUTS III, IV Agricultural Paying Agency (APA)
(remark: available partially (investment measures on national level or NUTS III))
http://www.apa.sk/index.php?na vID=353
Data on obtained level of support from other EU and national programmes (total)
national CCA - coordination of programming on national level
(remark: each support programme has its own web site and own monitoring/ reports)
http://www.nsrr.sk/operacne- programy/
Main enterprise/holding structural characteristics, e.g. including:
Production (value and structure, incl. crops(*), livestock(*); marketable and non-marketable; high quality value, etc.); (*)Agricultural Farms only
national, NUTS III, IV National Statistical Office www.statistics.sk
Turnover/Revenues (level and structure) national, NUTS III National Statistical Office www.statistics.sk
GVA (see:
http://enrd.ec.europa.eu/app_templates/filedownload.cfm?id=84053
593-C697-FF89-ED5C-51797D9754FD; and
http://enrd.ec.europa.eu/app_templates/filedownload.cfm?id=8D793 2B6-D3A1-95CC-8A26-14ABFAB6D327)
national, NUTS III, IV National Statistical Office www.statistics.sk
The level and (%) structure of non-agricultural GVA (e.g. tourist vs. non-tourist origin)
national, NUTS III, IV National Statistical Office www.statistics.sk
Profits and Gross Margins national, NUTS III National Statistical Office www.statistics.sk
Household income (agricultural and non-agricultural) national, NUTS III, IV National Statistical Office www.statistics.sk Area (structure of land ownership; and land use structure; of which
NATURA2000 sites and river basin areas affected by the WFD; first afforestation, etc.)
National Statistical Office, CD Ministry of Agriculture and Regional Development (MADR) of the SR, National Agriculture and Food Center (NAFC) -Research institute for agriculture and food economics (RIAFE), Water Research Institute (WRI) , National Forest Center (NFC) SR, NAFC - Soil Science and Conservation Research Institute, Eurostat
Agricultural Farms (total population – beneficiaries and non-beneficiaries )
Population size Owner Weblink (URL)
Employment (i.e. agricultural, non-agricultural; family, non-family; educational level; age structure; etc.)
national, NUTS III, IV National Statistical Office www.statistics.sk
Labour costs (hired labour) national, NUTS III, IV National Statistical Office www.statistics.sk
Value and structure of capital (buildings, machinery, livestock) national, NUTS III, IV National Statistical Office www.statistics.sk
Value of investments national, NUTS III, IV National Statistical Office www.statistics.sk
Milk quotas, etc. Agricultural Paying Agency
Yields (for Agr.Farms by main crops and livestock) national, NUTS III, IV National Statistical Office, MADR www.statistics.sk
Use of fertilizers The Central and Testing Institute in agriculture in
Bratislava, Slovakia (ÚKSÚP)
Purchased feed National Statistical Office, MADR www.statistics.sk
Total costs National Statistical Office www.statistics.sk
Data on area under successful land management contributing to:
biodiversity and High Nature Value farming/forestry NAFC - Soil Science and Conservation Research Institute
+ National Forest Centre (NFC)
Ministry of environment/State nature protection of SR MADR SR
protection of wildlife species or groups of species National Forest Centre, Ministry of Environment/State nature protection of SR
safeguarding endangered animal breeds and plant varieties NAFC-Plant Production Research Institute; NAFC-Animal
Production Research Centre, Slovak University of Agriculture
land cover types corresponding to HNV farmland (i.e. semi-natural pastures and meadows; traditional orchards; mosaics of low- intensity crop types; fallow land in low intensity farming systems; natural and semi-natural forests);
NAFC - Soil Science and Conservation Research Institute, NAFC - Plant Production Research Institute, National
Forestry Center
MADR SR