According to Scriven (1991), evaluation is the systematic determination of the quality, value, or importance of something. The ‘something’ can refer to a whole entity, or it can refer to aspects or components of an entity (Davidson 2005). The types of things that can be evaluated are broad and may include projects, programs or organisations, personnel or performance, policies or strategies and products or services (Davidson 2005). Evaluations are typically undertaken for one or two major reasons, namely, to find areas for improvement and/or to produce an assessment of overall quality or value, which is generally conducted for reporting or decision- making purposes (Davidson 2005). Furthermore, in a review of evaluation literature, Christiansen (2004) found that one of the reasons for conducting evaluations was for accountability, and that the purpose is to use the evaluation to underpin support for continued investment.
Robson (2000) claimed that there was a range of purposes for conducting an evaluation, and likely questions that would be asked in relation to each purpose (see Table 2.). In regard to the present study, the most pertinent purpose would be to assess the outcomes of a program. Correspondingly, a likely question that could be asked in regard to event evaluation is: What is the social impact of an event on the host community? Further, from the perspective of an STO that provides support for the event, a question might be: Is it worth continuing to support the event?
Table 2.1 Purpose of Evaluations and Likely Questions Posed
Purpose Likely Questions
To find out if the needs are met What should be the focus of a new program?
Are we reaching the target group?
Is what we provide actually what they need? To improve the program How can we make the program better (for
example, in meeting needs; or in its effectiveness; or in its efficiency)? To assess the outcomes of a program Is the program effective (for example, in
reaching planned goals)?
What happens to clients as a result of following the program?
Is it worth continuing (or expanding)? To find out how a program is operating What actually occurred during the program?
Is the program operating as planned? To assess the efficiency of a program How do the costs or running the program
compare with the benefits it provides? Is it more (or less) efficient than other programs?
To understand why a program works
(or doesn’t work) They are unlikely to seek answers to this – but such understanding may assist in improving the program and its effectiveness
Source: Robson (2000, p. 10)
In terms of how an evaluation is undertaken, Davidson (2005) described two steps that are involved in conducting an evaluation. The first step is to establish a clear understanding of what is being evaluated and what types of questions need to be answered. Following this, the second step is to identify the relevant values, collect appropriate data, and then systematically combine the values with the descriptive data to convey answers to the key evaluation questions that were asked (Davidson 2005). Similarly, Robson (2000) maintained that a high-quality evaluation required a well thought-through design phase as well as the collection, analysis and interpretation of data.
2.2.1
Evaluation Theory
Evaluation theory has, over time, progressed through four distinct phases (Davidson 2005; Robson 2000; Shadish, Cook & Leviton 1991), with each successive generation of evaluation representing a step forward in terms of the content and level of sophistication of evaluation (Guba & Lincoln 1989). Robson (2000) maintained that the early focus of evaluation was on experimental or quasi-experimental types of evaluation, and that the second generation was where evaluations were used in the actual process or decision-making contexts. Robson (2000) suggested that this change represented a shift from a knowledge-driven approach to a user-led approach. In addition, Robson (2000) referred to the third generation as a paradigm war between a number of evaluation authors, whilst the fourth generation has been labelled as naturalistic and constructivism, or responsive constructivist evaluation (Guba & Lincoln 1989). It was suggested that fourth generation evaluation outcomes do not represent the way things are, rather, they are constructs, which are formed by multiple actors in order to make sense of the world around them (Guba & Lincoln 1989). As such, evaluation findings are literally created through a process that includes the evaluator and other persons involved in the evaluation. The outcomes of this process may be regarded as the ‘realities’ of the case (Guba & Lincoln 1989).
2.2.1.1 Outcome Evaluations
An evaluation of the outcomes of an entity is a more open and exploratory style of assessment compared to an evaluation of the achievement of prespecified goals, and is a frequently requested evaluation task (Robson 2000). In outcome evaluations, the task is largely the selection of appropriate outcome measures, rather than the use of a particular research method. Moreover, the measures will be specific to the particular entity being evaluated (Robson 2000).
One of the emerging streams in the evaluation literature has focused on assessing the outcomes of organisations such as businesses. Love (2001) claimed that the driver for change in evaluation was the need for increased accountability, effectiveness and efficiency, and that this had sparked a measurement revolution for many organisations. Similarly, Robson (2000) suggested that in the current age there is an
increasing need for accountability and concern for value for money. Moreover, there seems to be a requirement to monitor, review or appraise virtually all aspects of the functioning of organisations in both public and private sectors. Love (2001, p. 438) concurred and stated that ‘the bottom line for organisations in all sectors is the clear demand to measure outcomes and use both quantitative and qualitative data to tell compelling “performance stories” about how well their strategies have worked’. In addition, Love (2001, p. 438) maintained that the trend towards evidence-based practice and the guideline movement were specific facets of the measurement revolution, and that this was ‘the cornerstone of the strategy being used in many sectors to reduce variability of services, improve quality, measure outcomes and reduce costs’. Finally, from an international perspective, there is potential for evaluation to contribute to solving problems such as improvements in the health of the environment and sustainable economic development (Mertens & Russon 2000).
In short, evaluation is the determination of the quality, value or importance of an entity and this evaluation can relate to the measurement of the outcomes of the entity. In the context of this research, the ‘something’ that will be evaluated is a special event. More specifically, the aim is to develop a set of indicators to evaluate the outcome of a special event in regard to its TBL impact. As such, the next step is to explore the concept of the TBL, which is the focus of the main section of this chapter.