B.1
Introduction
The objective of the econometric study conducted as part of this evaluation is to assess the impact of the FSP on production and incomes. The scope of this impact analysis is necessarily constrained by the available data and the extent of the inputs allocated for this study. In light of these constraints the study focussed on attempting to answer the following questions:
1. Has the introduction of FSP increased maize crop productivity (kilograms produced per cultivated hectare) amongst those farmers benefiting from the programme?
2. Has any increase in maize crop productivity come at the cost of a decrease in productivity of other key crops (i.e. diversion of inputs such as prime fertile land away
from non-maize crops)?34
3. Has the introduction of FSP increased total crop income (sum of the market value of all crops produced) amongst those farmers benefiting from the programme?
4. Has the introduction of FSP increased total crop income net of the market value of
fertiliser used amongst those farmers benefiting from the programme?35
A recent study by Xu et al (2009) suggests a potential concern that, due to poor targeting, the FSP may in fact be crowding out private purchases of fertiliser amongst farmers who would have used fertiliser in any case, i.e. regardless of whether or not they were in the programme. One implication of this is that the FSP may simply represent an inefficient form of wealth redistribution, in the form of an in-kind transfer (i.e. fertiliser), to relatively better off farmers with no associated gains in aggregate agricultural productivity.
For this reason it would have been interesting to assess: (i) whether FSP beneficiaries who previously had not been using fertiliser are significantly more likely to have taken up use of fertiliser than they would have been in the absence of the programme; and (ii) whether FSP beneficiaries who had already been using fertiliser pre-treatment are more likely to have increased fertiliser use than they would have been in the absence of the programme. However, this analysis was beyond the scope of this study. Instead this issue is assessed indirectly, by estimating the comparative impact of the FSP for those farmers that already used fertiliser in the baseline year. If the FSP impact is found to be smaller for these farmers, then it implies that the FSP impact is greater for farmers that are not using fertiliser initially. This in turn has strong implications in terms of the most beneficial targeting of the programme.
A further concern is that FSP fertiliser often arrives late, reducing significantly its effectiveness. In order to investigate this, by examining the impact of the FSP programme
34
In addition to being as interesting policy question in itself, question 2 highlights a concern that beneficiaries might divert their more productive land to grow Maize as a result of the FSP, and unless we take this into account we may overstate the combined productivity impact of the inputs that comprise FSP pack.
35
The idea behind this question is to investigate whether the positive effect of FSP outweighs its cost. To do so requires looking at total expenditure on all inputs, and hence change in profit, as inclusion in FSP may be correlated to changes in expenditure on other inputs. However, information on total input expenditure is not available so this analysis was limited to assessing the impact of the programme on total crop income net of fertiliser.
when the fertiliser arrives on time, the same questions are addressed considering just those households who report receiving the fertiliser on time.36
B.1.1 Overview of the methodology
The econometric analysis undertaken addresses these questions by employing Propensity Score Matching (PSM) methods. This non-parametric technique matches FSP beneficiaries with a counterfactual comparison group. The propensity score matching estimator is applied to a difference-in-difference (diff-in-diff) specification and the differences between FSP beneficiaries and the counterfactual group in the changes of the outcome variable (e.g. post- treatment versus pre-treatment maize productivity) are computed. The PSM methodology therefore yields average treatment on the treated (ATT) estimates of FSP participation for changes in the outcomes of interest.
Three alternative variations of the Propensity Score Matching approach were employed for
this study.37Each of these methods is discussed below.
Method 1: Standard PSM Approach
The standard PSM approach is to match on pre-treatment characteristics. What this means in practice is that for each FSP beneficiary in the follow-up period a non-FSP household, or group of households, is identified that were sufficiently similar to FSP users prior to the introduction of the programme, i.e. in the baseline period. This group of pre-treatment comparable non-beneficiaries is referred to as the control group. The beneficiaries are referred to as the treatment group. Intuitively, the control group is identified such that ex ante they would have been expected to be just as likely to have been a beneficiary in the follow- up period as the treatment group. In theory, and under certain assumptions (discussed below), this means that any difference in the change in key outcome measures observed between the treatment and control groups can be directly attributed to the impact of the programme.
One weakness with the application of this approach for this study is that a household is identified as being a FSP beneficiary when reported as a beneficiary in a single survey year (i.e. in the follow-up period). However, the beneficiary status of a household might vary in the years between the survey years. Similarly, some non-FSP households might have benefited from the programme during non-survey years. This uncertainty in the continuity of treatment status will lead to a downward bias in our FSP impact estimates, since it will reduce the average cumulated benefits among FSP beneficiaries and increase the cumulated benefits among non-FSP households.
A second, more substantive, concern relates to the extent to which the control group actually represent a credible counterfactual. A crucial assumption in propensity score matching methods is the ‘ignorability of treatment’ condition, which states that treatment and control not only are observationally similar but also do not differ systematically in their unobservable characteristics. Under the assumption of ‘ignorability of treatment’, any difference in the change in key outcome measures between the treatment and control groups can be directly attributed to the impact of the programme (Heckman, Ichimura, and Todd, 1997 and Smith
36
We define such households as those reporting that both basal and top fertiliser were available from the FSP programme at the time that they were needed.
37
Further details of the PSM difference-in-difference methodology, and its limitations, are discussed in more depth in section B.2.3.
and Todd, 2001, 2005). This is a strong assumption as it implies that farmers are similar in their unobserved attributes, such as abilities, social and political networks, health, etc.
Our difference-in-difference specification allows this condition to be relaxed somewhat, as it controls for any time-invariant unobserved farmer characteristics (Heckman, Ichimura, and Todd, 1997). However, this methodology will still be subject to biases arising from any systematic differences in time-varying unobservables across treatment and control. For example, such bias would arise if FSP farmers have higher productivity ability – that is they are able to make the most of a new technology and obtain higher maize yields – than non- beneficiaries. To the extent that FSP beneficiaries have higher ability than non-FSP farmers, and therefore more likely to experience relatively higher productivity growth even in the absence of the FSP, then the standard PSM approach is very likely to overstate the FSP impact.
This shortcoming is addressed by applying the following two alternative methodologies. Both are based on the assumption that higher farmer ability might lead to a higher rate of asset and capital accumulation over time. Controlling (a) for these changes directly and (b) for post-treatment characteristics is therefore an attempt to isolate the FSP effect from any unobserved ability bias.
Method 2: Standard PSM approach controlling for changes in crop land, net income and assets
This method attempts to control for systematic variations between the treatment and control groups in three key channels that drive productivity growth, namely increases in land area available for crops, total household income and the value of the household’s productive assets.
This was achieved by estimating the relative growth (difference-in-difference) in crop land, net income and assets in turn, for beneficiaries as compared to the non-FSP households. The relative growth of each of the key impact outcome measures (maize yield, crop income, etc) was then regressed on the measured relative growth of crop land, net income and assets. This method therefore delivers an estimate of the impact of the FSP once systematic variations between the treatment and control groups in the growth of crop land, net income and assets have been removed.
Ideally, all key factors that drive productivity growth should be controlled for. However, due to time limitations it was only possible to account for these two channels, so if other factors are involved (and which also exhibit systematic trend differences between treatment and control groups) then this method may still overstate the FSP impact.
Method 3: Modified PSM approach (matching on post-treatment characteristics)
This method modifies the PSM methodology by matching on post-treatment characteristics and not on pre-treatment characteristics, as is standard in the PSM literature. For the alternative matching the same categories of characteristics as those used for the standard PSM method were used. By matching exclusively on post-treatment household observable characteristics, it is hoped to more systematically capture the higher asset accumulation and income path on which higher ability farmers are placed. However, by doing so, this method will also remove a part of the FSP impact, to the extent that FSP benefits also resulted in changes in the post-treatment household characteristics. This approach may therefore understate the FSP impact.
B.1.2 Data
Box 8.1 describes the sources of data that were available for the purposes of this evaluation and factors influencing their relevance to the econometric analysis of the FSP. Data from three questionnaires were combined to undertake the analysis. These were the 1999/2000 Post Harvest Survey (PHS) and its Supplementary Surveys (SS) for 2001, 2004 and 2008. The 2001 Supplementary Survey primarily refers to the 1999/2000 season, the 2004 Supplementary Survey to the 2002/3 season and the 2008 Supplementary Survey to the 2006/7 season.