B.2.3.1 Unobserved heterogeneity and systematic post-treatment variations between beneficiaries and non-beneficiaries
While the PSM approach does go some way into correcting for differences in observables between assignment groups, it does not address issues of unobserved heterogeneity. In other words, any analysis based on propensity score matching is open to the criticism that FSP beneficiary status might be driven by systematic but unobserved pre-treatment differences between FSP and non-FSP households. In particular, the highly selected nature of FSP participants combined with the loosely implemented FSP eligibility criteria imply that PSM methods will be exposed to reservations on the adequate comparability of the matching control group.
PSM difference-in-difference methods are able to control for those differences to some extent, but will nevertheless be exposed to time-varying differences. This would be the case, for example, if FSP households have unobserved characteristics that mean their productivity growth trend would have been higher than other households even if they had not benefited from the programme. Such difference in growth trends could arise if, for example, FSP beneficiary households are, on average, better connected and/or are more capable farmers and are therefore systematically better able than non-beneficiaries to access and adopt new innovations and to translate these into productivity gains. In this case the standard PSM difference-in-difference methodology will tend to overstate the FSP impact.
Another source of concern is with regard to the bias resulting from time-varying factors influencing the outcome measures that are correlated with FSP status even when unobserved heterogeneity is not a problem. This could arise, for example, if beneficiaries were systematically excluded from initiatives introduced since the FSP started, such as government- or NGO-funded rural-credit or irrigation programmes, that have an impact on productivity. Such a scenario may be quite likely in a context where resources available for supporting rural livelihoods are scarce, and therefore new programmes may be expected to target households not already benefiting from existing forms of support such as FSP. In this case the PSM difference-in-difference methodology will tend to understate FSP impact since
the measured affect is being confounded by the impact of these other initiatives amongst non-beneficiaries.
Conversely, the opposite phenomenon could occur such that FSP receipt might be positively correlated with other government program benefits. For example, preferential access to land under new land expansion programs, preferential access to selling maize at FRA depots, irrigation investments, access to agricultural credit, etc. In this case PSM difference-in- difference methodology will overstate FSP impact.
Systematic post-treatment variations might also arise if, to the extent that farmers use different plots for different crops or even inter-crop, FSP fertiliser and maize seed packages tend to be used in higher quality plots while the pre-treatment conventional maize seeds would have been planted in lower quality plots. In other words, in response to treatment the beneficiaries may reallocate inputs between different crop types. If this does happen, and is associated with better quality inputs or input combinations being allocated to maize, then the PSM difference-in-difference methodology will overstate the impact of FSP on maize productivity, since some portion of any increase in productivity is simply due to allocating better quality plots rather than the FSP itself, thus confounding the results.
This problem of systematic post-treatment changes in plot quality could potentially be addressed if a panel of fields could be constructed. However, such data was not available for this study, although going forward it is potentially possible that such a panel could be constructed on the basis of the plot diagrams collected in every round of the SS. An alternative approach would be to consider the impact of FSP on total crop productivity, which would net out any shifts in the post-treatment allocation of inputs between crops. The drawback of this approach is that it is not obvious how to aggregate total crop output across different crops.
To assess this issue, this study estimates the effect on the FSP on the yield of another major crop, groundnuts. A similar analysis was carried out on the impact of FSP on the yield of millet though the sample size was not sufficient to yield any meaningful results. Whilst not a perfect assessment of the impact of the FSP on non-maize production, these crops nevertheless provide an indication as to whether the FSP might be leading farmers to allocate their resources different across crops.
B.2.3.2 Heterogeneity of treatment
It is clear that there is quite a wide variation in the experience of FSP beneficiaries in terms of the treatment they receive. Most obviously, and as discussed above, treatment is not constant across all years which means that current FSP beneficiaries have not necessarily been in the programme for each year it has been operating, and conversely some ‘current’ non-users will have benefited from the programme in previous years. Since the impact measures will be confounded by control group households that may have in fact have benefited from the programme in at least one year prior to the ‘current’ period and diluted by treatment households that may in fact have only experienced one year of treatment, this form of treatment heterogeneity will imply a downward bias on the impact estimates.
Another form of treatment heterogeneity stems from the fact that, although the FSP pack is designed to for use by just one farmer, in practice some groups of farmers are reported to be joining together to purchase and share individual FSP packs. Furthermore, problems in the implementation of the programme have been reported, with some beneficiaries receiving their packs too late (i.e. after the planting season). It is reasonable to suppose that a household that is sharing its FSP pack and/or receiving it late would not be expected to experience the same impact as a household receiving the FSP on time and in-full. Therefore
if many beneficiaries are receiving very low or mistimed “doses” of treatment in this way, this will lead us to under-estimate the impact of FSP, since many of the beneficiaries are not receiving treatment in such a way that will actually increase crop productivity.
Whilst the scope of this study means that it has not been possible to investigate the impact of different levels of treatment (i.e. of farmers using the fertiliser less intensively than intended), this study does consider the impact of the whether the fertiliser is delivered on time.
B.2.3.3 Non-decomposability of the treatment impact
The FSP “treatment” is comprised of a pack containing not just fertiliser but also improved maize seeds. From a policy perspective it would clearly be interesting to understand the relative contribution to overall impact of each of these two components. In a difference-in- difference OLS setting, it might be possible to attempt to disentangle the fertiliser effect from the improved seeds effect by controlling for the changes in inputs and explicitly include fertiliser use at period t. To the extent that there is a remaining effect of FSP treatment, it would suggest that the improved seeds are effective.
However, to the extent that beneficiaries tend to apply the fertiliser and seeds together, as advised by the FSP programme, variation between the two inputs is unlikely to be sufficient for the two channels to be disentangled. For this reason, this decomposition of treatment impact is not undertaken for this study. However, it is important to recall that the impact identified below may not hold for the impact of hybrid seeds or increased fertiliser individually.
B.2.3.4 External validity of the impact results
In terms of the implications of this study for policy going forward, the issue of external validity is very important, since this directly relates to the extent to which the results are relevant for forward-looking policy making. For example, if current beneficiaries differ systematically from those households that would benefit from the programme should it be expanded, it could imply that current beneficiaries are not representative of future beneficiaries and therefore that the ATT effect differs from the expected ATE going forward. In other words, the measured benefits to current beneficiaries (ATT) are not representative of the average benefits (ATE) when FSP participation is expanded to a wider population.
Similar concerns of the external validity of the results of the analysis arise from the fact that current treatment implementation may not be representative of future FSP implementation in an expanded programme. On the one hand, poor implementation of the current programme (low/mistimed dosage, sharing packs, etc) may be improved, which could imply that the currently observed impact may underestimate expected impact of an expanded programme. Conversely, implementation quality may in fact deteriorate as the programme is expanded, which would mean the impact results from this study would over-estimate impact of the programme going forward.
In fact there has been significant change in FSP intervention modality for the 2009/10 production season. The size of the pack was halved and the number of participants doubled. As a result application rates and productivity impacts are both likely to be reduced.