Construcción e instalación del equipo de prácticas de medidas

The aim of this literature review is to identify the characteristics in each of the informing disciplinary areas which may constitute obstacles, difficulties and disincentives to evaluating effectiveness. This review will provide a basis for identifying the synergies between the key research challenges which characterise effectiveness evaluation across the informing disciplinary areas.

1.12.1 Key characteristics and research challenges in the informing disciplinary areas – the small business context

The small business sector is most critically characterised by heterogeneity and this has implications for the type of training and support developed and delivered to those in small business. Evaluation research in the small business context is characterised by challenges in the

areas of the complexity of the phenomena, context, methodology, data collection, paradigm location and measurement which arise out of this heterogeneity.

1.12.1.1 Complexity and variability in small business

In discussing the complexity and heterogeneity of the small business sector, Atterton says:

We talk about the small business sector as if it is some sort of market niche and I do not know how 96 per cent of anything can be a market niche. It is a massively heterogenous group and I think we could do far more in terms of segmentation: which segments we want to work with and how we develop the capability to work in that sector (Atterton 2002, p.970).

Assistance to the small business sector can therefore be characterised by, in Atterton’s terms, the need for segmentation. There is widespread acknowledgement of the “uniqueness” of each business in terms of size, the type of business engaged in, profit and turnover, whether home or office-based, the industry sector in which they operate, in the products and services produced, in the processes and level of technology used, and in the specific community and business

environment in which they are located (Tolentino 1998, Devins & Gold 2000). This complexity and the multiplicity of intervening contextual variables which may influence outcomes has implications not only for practice but also for exploring the concomitant variability in effectiveness outcomes.

1.12.1.1.1 The appropriateness of competency-based approaches in small business

In the United Kingdom, the trend in small business research is toward a competency-based approach to developing learning frameworks for owner-managers in small business. This approach whereby learning is related to broad external competencies is however criticised by Gibb who suggests that generic competency standards as a basis for entrepreneurial learning is inconsistent with the research on learning requirements for small business. The entrepreneurial literature strongly indicates a need and preference for individualised, contextualised and experiential learning (Gibb, A.A. 1997). This was also confirmed in the work of Devins and Gold (2000). “Cracking the Tough Nuts: mentoring and coaching the managers of small firms”, an exploration of mentoring of the managers of small firms who participated in a program called Building Management Competencies. A case study approach was used to examine and evaluate the program’s impact on 20 organisations working closely with their Business Coach based on generic management competencies. Devins and Gold found that:

The crucial point about all the examples is their unpredictable path and their lack of connection to the predicted package of resources and activities that had been developed in advance of the programme. There was only one case which has followed what might be referred to as the ’typical model’ of business support (Devins & Gold 2000 p.254).

Devins and Gold found that the model of business support they applied did not provide for the heterogenous and diverse engagement with the support provided.

Just as the characteristics of small business have implications for practice, so too are there implications for evaluation research.

1.12.1.2 Quality of data in small business research

1.12.1.2.1 Nature of data

Much small business research gives rise to self-report data. Storey (1998) identified problems with bias and error in self-report research methods in small business in that “.. some

entrepreneurs will overestimate the impact of the initiatives … [while others] .. are likely to underestimate the contribution of policy by claiming that any improvements in their business reflected their entrepreneurial skills, rather than public money” (p.20). Nisbett (1977) suggested that findings using self-report data may be contradicted by findings using other methods of data collection, so data quality is a key methodological challenge in small business research.

1.12.1.2.2 Sampling problems

Sampling difficulties are a major issue with much small business research largely due to the heterogeneity of the sector. The segmentation approach advocated by Atterton has implications for research in that any sample based on a particular “segment” is potentially atypical of the small business sector more generally and this impacts on the validity of inferences made to the broader small business population.

The heterogeneity which characterises small business can also make it difficult to construct matched samples (Curran & Storey 2000). This means that experimental work comparing outcomes of a group which has accepted business support with a control group that has not, is problematic. Evaluating effectiveness which is grounded in positivist assumptions leads to a difficulty in establishing causality between an intervention and effectiveness.

Small sample size is another issue frequently cited as a sampling difficulty in small business research. This is attributed to a range of factors including poor take up of support services (Curran & Blackburn 2001, Curran 2000), low response rates (Curran & Storey 2000), and the difficulty of accessing sufficiently large sampling frames (Curran & Blackburn 2001). This means that in experimental research which makes claims relying on statistical validity, a sample may not, in Curran and Storey’s terms, meet “statistical criteria for establishing validity” (2000 p.17). With the heterogeneity of the small business population, a very large sample would be needed to ensure representativeness across a range of variables that is often not available.

Curran and Storey (2000) point out that while small sample size is not necessarily of itself a difficulty, the issue of response bias which may follow from small sample size is potentially a problem. They identify the most common biases in small business research as firm size bias (that is, smaller firms have been shown to be less likely to respond than larger firms) (Goffee and Scase 1995 cited in Curran & Storey 2000) and sector bias (that is, firms in some sectors are more likely to respond than others (Curran & Blackburn 2001). If the individuals or firms who respond have, in Curran and Blackburn’s terms, “had a more positive experience than those who do not” this may also result in response bias and therefore impact on the external validity of the research findings (Curran & Blackburn 2001 p.61).

1.12.1.2.3 Bias

While potentially having a positive impact on outcomes in practice, administrative and self- selection are problems which arise in evaluation research in relation to small business (Curran & Storey 2000). Curran and Storey define administrative selection as occurring “when only a proportion of the firms/individuals which apply to join [a] program are selected for inclusion” (2000 p.13). They define self-selection as occurring “when certain types of firms apply to join/participate in particular programmes” (p.13). The issue is that when these forms of selection occur, the outcomes of an intervention may not be representative in that the selection process may have involved firms or individuals who are atypical in some way. That is, the qualities that led to the individuals nominating or being selected for participation in a program may produce causal ambiguity in that these qualities, rather than the assistance program, may have contributed to the outcomes observed or reported.

Generalisability in small business research then is always open to challenge on the basis that the representativeness of the sample which supposedly represents the segment may be challenged therefore compromising generalisations to the particular small business segment and the small business sector more generally.

1.12.1.3 Measurement difficulties in small business research

1.12.1.3.1 Definition of concepts

The problem of adequately defining concepts and constructs in small business research is highlighted by Curran and Blackburn (2000). Failure to define concepts such as small business and small business owner, they suggest, may make comparability between studies difficult. Buelens et al. (2005) also discuss the need to establish the validity of constructs by ensuring that the measures being used are in fact appropriately operationalised, sufficiently comprehensive, confirmed, tested, and indicative of the constructs being used in a study, all of which threaten the measurement of effectiveness in the small business context.

1.12.1.3.2 Causality

Measurement problems are widely discussed in small business research. Storey’s study “Six Steps to Heaven” (1998) considers the evaluation of the impact of an assistance program. While it considers evaluation at the policy level, the methodological and conceptual difficulties which are discussed in Storey’s approach are also relevant to evaluation at other levels. Storey (1998) details the need to measure additionality with reference to deadweight and displacement, and discusses the difficulties with finding evaluators, politicians and policy-makers who will accept such issues as important to effectiveness evaluation.

Curran and Storey (2000) suggest that “it should be relatively easy to measure [additionality accounting for deadweight and displacement] .. but .. in practice, it is extremely difficult” (2000 p.11). This is because of the diversity of contextual variables which must be controlled to produce small business research with high internal validity. One of the major threats to internal validity is the failure to control for the influence of external or intervening variables which means outcomes cannot be attributed solely to the assistance provided to a small business (Buelens et al. 2005). This creates what is another major challenge in small business research - that of establishing causality between an intervention and its outcomes - because of the diversity of contexts into which the assistance is delivered, and the impossibility of controlling for all relevant variables.

Gibb confirms these challenges about small business training interventions when he states:

In the light of the substantial research that has been undertaken into cost benefit analysis of training there must be considerable doubt as to whether a definitive answer could ever be found to the question of payback on training (Gibb A.A. 1997 p.13).

Given the difficulties of, in Gibb’s terms, finding such definitive answers, the question then becomes “how can claims around structured e-mentoring effectiveness be substantiated?”, and this is the subject of this study.

1.12.1.3.3 Capturing intangible benefits

The difficulties with measurement are symptomatic of a broader debate within small business research around the challenge of measuring intangible impacts. Lenihan and Hart’s struggle with how to “calibrate less tangible impacts into deadweight estimates” (Lenihan & Hart 2004 p.10) sits alongside Wright and Tao’s discussion around the issue of “hard” and “soft”

measurement (Wright & Tao 2001) which is played out in the literature as the relative merits of quantitative and qualitative approaches to small business research. This debate is based on an acknowledgement that benefits may be difficult to quantify in econometric or other terms. Small

business research has in the past been dominated by quantitative methods which are grounded in positivist assumptions based on the idea that reality can be explained and knowledge and truth revealed by using quantitatively-based experimental methods. In “Six Steps to Heaven” (1989) Storey asserts that sophisticated evaluation relies on objectives being specified in a quantitative manner in the form of targets (p.4). “It is then necessary,” he says “to compare the assisted firms with groups of firms not assisted by the policy” while all other influences are held constant (p.21). Storey does not address how experimental methodologies assist with understanding or quantifying, in his terms, the “unobservables” (Storey 1998 p.27).

1.12.1.4 Research paradigm – location of this study in relation to existing research

The present trend in evaluation of intervention programs for small to medium enterprises (SME) is a return to goal-orientated approaches. In the Organisation for Economic Co-operation and Development (OECD) report “Evaluation of SME Policies and Programmes”, Storey (2004) advocates the use of approaches which specify goals, quantify targets and evaluate program success in terms of achievement of these goals. Storey’s model uses a monitoring/evaluation continuum which both contrasts and has parallels with a transdisciplinary model of program evaluation set out by Scriven (1993).

Scriven elucidates some of the limitations of the goal-orientated approach which implicitly underpin Storey’s methodology. Principally, Scriven suggests that such an approach potentially fails to take account of what he refers to as shortfalls, overruns and side effects. Scriven (1993) rejects the idea that program evaluation should simply involve assessing the attainment of predetermined goals. Such an approach may neglect important information on the effectiveness of a program. He says of side effects:

Side effects are often the main point ... Side effects were a latent killer for a literal interpretation of goal achievement evaluation. They cannot be ignored because they may require the abandonment of an otherwise successful program or the salvation of an otherwise unsuccessful program. But it is hard to design an investigation to find them, since they are, more or less by definition, unanticipated. The only systematic methodology for detecting side effects is the goal-free approach (p.49).

Scriven goes on to suggest that evaluation with reference only to a program’s goals potentially ignores what he terms “absolute values”, cost analysis, generalisability and comparisons – that is, could the same outcome have been achieved more affordably or with fewer negative side effects? He also notes that some program goals may have different relative importance and that there may be varying levels of success for a range of these goals potentially creating a complex set of data/results which the program evaluator must effectively judge, rank and synthesise. The difficulty with measuring effectiveness exclusively in terms of program goals as prescribed by

Storey is that it effectively values at zero the intangible, unanticipated “side effects” and evolving benefits.

Storey proposes a “Six Steps” taxonomy for evaluating SME assistance programs. The steps include Step 1 - Take up of schemes, Step 2 - Recipients Opinions, and Step 3 - Recipients views of the difference made by the Assistance. These three steps are referred to by Storey as monitoring. The next three steps include Step 4 - Comparison of the Performance of ‘Assisted’ with ‘Typical’ firms, Step 5 - Comparison with ‘Match’ firms, and Step 6 - Taking account of selection bias. Steps 4, 5 and 6 are referred to as evaluation. Storey goes on to say that using a control group, applying statistical methods to account for the influence of variables other than those being studied and measuring against specific and pre-set targets ensures appropriate evaluation is occurring. In his proposed evaluation framework, Storey suggests that measuring use and asking users about their perceptions of value can be referred to as monitoring because they do not take into account an intervention program’s objectives. A summary of the binary oppositions created by Storey’s stance are set out in Table 1.

Table 1 – Summary of characteristics of monitoring compared with evaluation according to Storey 1998

Monitoring Evaluation

• undertaken by cheap and cheerful brigade

• happy sheets

• sloppy analysts

• analysts without integrity

• methodology – self-report (includes selfish and self- interested as well as truthful)

• only building blocks for evaluation

• simple

• targets – anything it happens to hit

• selection methods ignored

• not Heaven

• careful

• accurate

• sophisticated

• serious research community

• methodology – must necessarily involve a control group

• statistical methods

• objectives which should be quantified and become explicit targets

• approach receives “heavyweight” support

• controlled for selection

• Heaven

What are the implications of the conceptual separations between monitoring and evaluation set out in Table 1 for effectiveness evaluation in the small business context?

Storey’s approach is limited in that it is prescriptive about methodologies to be used to evaluate effectiveness. The most significant limitation of Storey’s analytical framework is that outcomes which cannot be researched using quantitative methods are implicitly valued as contributing less to an understanding of effectiveness than those which can be measured in quantitative terms.

While controlling for selection and including a control group to conduct an evaluation may be seen as ideal to researchers looking to prove or disprove the influence of policy or program interventions, the characterisation of evaluation methods which do not use a control group or use statistical techniques as less accurate, careful, rigorous or sophisticated monitoring is open

to challenge. Since mentoring and e-mentoring are set within a complex system of internal and external variables, experimental design is less likely to be useful in advancing the field. Research using other methodologies can extend knowledge about the effectiveness of small business interventions in different ways. Storey’s work can be interpreted as reworking or reiterating a tension which runs throughout much of the entrepreneurial learning literature – between hard and soft measurement and ultimately between positivist and post-

positivist/constructivist methodologies. Storey’s analytical framework ignores the large body of research from the 1960s and 1970s which considered the virtual impossibility of ever being able to find a definitive causal relationship between a training intervention and program outcomes. (Gibb, A.A. 1997). Storey’s framework potentially “values at zero” the rich data which may be obtained from alternative methodologies such as self-reported perceptions of value, tangible and intangible benefits, evolving benefits, and unanticipated benefits or side-effects. Evaluation of the effectiveness of structured e-mentoring in the small business context should not be restricted by neglecting or stigmatising understandings gained by approaches to measurement beyond those supported solely by quantitative data and grounded in positivist assumptions.

1.12.1.4.1 Implications of adopting a positivist paradigm in small business research

Storey’s approach to small business research can be considered in the context of the growth and contradictions in the paradigms through which knowledge in the social sciences can be

advanced.

Science philosopher Kuhn (1970) conducted a study of the value systems of scientists. Kuhn comments that “quantitative predictions are [considered] preferable to qualitative ones” and that the methodological status hierarchy in science ranks “hard” above “soft data” where “hardness” refers to the precision of statistics. Qualitative data, then, carry the stigma of “being soft” (Kuhn 1970 pp.185-186). Storey’s approach to small business research privileges hard over soft data and quantitative over qualitative analysis in the way described by Kuhn.

Nissen discusses human-centred research which must necessarily capture the opinions, beliefs, attitudes and perceptions of the social actors involved (Nissen in Mumford et al. 1984 p.39). The methodological challenge to effectiveness evaluation is that all these share the characteristic of being difficult to measure. Weick (in Mumford et al. 1984 p.5) emphasises this point by suggesting that if these “non-measurables” are ignored, they are effectively valued at zero. Weick quotes Vickers as saying, “I recall times when I have criticised some forecast or estimate for omitting some variable which must obviously be relevant to the result and have been

omitting it, you have valued it at zero; and you know that is the only value it cannot have”