The summary by Cook et al. (2008 – as quoted at the end of Section 2.5) remains valid even when considering recent evaluation evidence. Evaluators suggest certainty around the impact of an intervention based on assessment time frames that are usually fairly short, and, far more concerning, not at all explained or justified. It does make “no sense at all” that, regardless of the nature of an intervention and regardless of its target group, there is hardly any discussion as to how timing was chosen. As explored in Section 2.4 there are calls for applying a longer time frame, however.
Of course, there are good reasons for how evaluation is currently undertaken and timed. Firstly, the interest in a particular scheme’s success fades over time, as the policy environment moves on. Policy-makers are keen to understand effects quickly. Funding for evaluations is usually awarded close to an intervention taking place, not years later.
Other reasons include the issue of access to data. Even if an evaluation is undertaken numerous years after an evaluation, the data captured as part of an intervention will usually not stretch much beyond the intervention period. Interest in capturing and maintaining data over numerous years is limited, as political and organisational changes may quickly render previous policies (and interest in them) obsolete. Capturing data retrospectively through surveys is very costly and relies on the willingness and ability of participants to remember their participation in the scheme of interest and provide the required insights into their business (and it still requires researchers to be able to identify a list of participants to survey)31. An alternative
approach would be for qualitative evaluation (Done et al., 2011), but that will always be limited to insights based on few firms, and possibly subjective data.
With the intention to explore long-term evaluation of non-financial business support this thesis also ran into data availability issues. Originally, for the purpose of this thesis, it was agreed with BIS to work with assistance data from the UK Regional Development Agencies (RDAs).
- 53 -
However, by the time the funding for this research was approved and the work on this thesis commenced, the RDAs were preparing for their March 2012 closure and their data had become unobtainable for BIS32.
In fact, the following analysis, whilst drawing on Business Link data, has been mainly funded to investigate the feasibility of long-term impact evaluation and to set out some methodological lessons for future work. With Business Link now reduced to a website, there is little appetite in government to understand the impact on firms originally assisted under the intervention over a decade ago. Other than the remaining website offer, the scheme has been replaced by Growth Accelerator (which in turn was set for closure at the time of printing this thesis). Attention has now turned to assessing its impact with the use of self-reported evaluation surveys undertaken independently (BIS, 2014c). However, lessons appear not to have been learnt as the short-term evaluation work points to a ‘positive impact’ of Growth Accelerator, yet such an assessment would not be ranked highly by the Maryland Scientific Methods Scale (as in WWG, 2014) or Storey’s Six Steps framework. In addition to the main critique around the use of the self-reported data for, in the case of the Growth Accelerator evaluation, economic impact analysis and gross value added estimates is that in this instance firms were essentially asked to provide estimates about anticipated effects in the future as a result of programme participation.
The danger that emanates from this lack of debate around the actual timing of an evaluation is that results are almost always presented with some certainty and confidence that “Scheme A was effective”, “Scheme B was not”. In response to the literature reviewed in this chapter and the observed short-term frames generally applied for evaluation – despite critical voices that long-term evaluation may be required – it needs to be empirically tested whether the accurate conclusion drawn from such evaluations would not be more accurately reflected by “Scheme A was effective for the one year period assessed, but there is no evidence to suggest
- 54 -
that this snapshot in time reflects the final truth about the effectiveness of the scheme (if there is such thing at all)”.
Such results would not only be quite a mouthful, but also unlikely to resonate well with research funders33. From their perspective, there are some obvious questions to respond to about the
claim that time matters for evaluation purposes – in line with the research hypotheses as introduced previously. In line with the contributions of this thesis as specified in Chapter One, what is required is:
1) Robust evidence demonstrating that short-term results do not necessarily tell the whole story.
2) A methodology that facilitates a workable approach to long-term evaluation.
The next chapter will introduce such methodology and the datasets that form the basis of my thesis, with the analysis and evidence undertaken presented in the chapters that follow.
33 Policy-makers met during the course of this PhD at various conferences and events were always
reasonably keen to get an absolute response, that is, “Scheme A was effective”, or “Scheme A was not effective”. It is also questionable whether research results that include such caveats would be cited with the limitations specified – often literature reviews omit such limitations.
- 55 -
Chapter Three
Developing a long-term evaluation framework
– Methodology & Data
The previous chapter highlights the apparent gap and desirability for long-term evaluation of business support programmes. This desire works against two main opposing forces. Firstly, political pressure will always mean a keen interest by government and other support bodies to quickly gain evidence on a scheme’s impact. Secondly, on the technical site, data availability presents a key obstacle for undertaking longitudinal impact assessments.
This chapter seeks to address the second of above-mentioned forces, and presents a methodological approach and the datasets required that allow such an evaluation of a business advice programme from intervention through to the present. In this thesis, the intervention in question took place in 2003.