The purpose of using Realistic Evaluation methodology to evaluate a programme is to establish whether or not it is better than previous ways of working, and whether it is preferable (Timmins and Miller, 2007). Realistic Evaluation takes into account the contexts within which mechanisms are working in order to produce the outcomes. This method of evaluation was particularly suited to evaluating the Treasure Project because it reveals the power within the different CMO configurations that might make up an effective programme. A more traditional, static method of evaluative research, however, might be only looking to confirm whether a programme was effective or not. Realistic Evaluation aims to reveal the inner workings of a programme and to use this to develop theory around ‘effectiveness’ so that this knowledge can influence practitioners’ future practice.
Phase 1 of the Treasure Project evaluation compared the Programme Specification (Table 3.5) with ‘regularities’ brought to light from a first set of ‘observations’. This chapter describes how Phase 1 informed the design of Phase 2 of the data collection. During this second phase key stakeholders, associated with successful schools, were asked to
participate in a realistic interview. These interviews sought to establish ‘what worked for whom under what circumstances’ in those schools where the project was viewed by the APCs as being particularly successful. The design of the interviews and the interim
refinement of the Programme Specification are described in this chapter. Chapter 6 goes on to describe the findings from Phase 2 and explain how the final adjustments of the
Programme Specification were made with regard to the new information about CMO configurations revealed by the realistic interviews.
5.2 Selecting the schools for Phase 2
Phase 1 compared the hypothesised CMO configurations derived from the Programme Theories outlined in the Programme Specification to the actual CMO configurations of the Treasure Project. The evidence from these findings was strong enough to support the suggestion that both Treasure Project interventions and the overall programme had been effective in promoting children’s mental health and emotional wellbeing and building capacity in schools. It could also confidently be acknowledged that both interventions and the overall programme had been more effective in Project Schools than in Non-Project Schools and that this difference in impact was linked to the multi-faceted, multi-level approach to mental health and emotional wellbeing, promoted through the participatory partnership with the APCs. In terms of the Programme Specification, it could be stated with reasonable confidence that the findings supported hypotheses derived from
Programme Theories 1, 3 and 4. However, hypotheses derived from Programme Theory 2 did not seem to be strongly supported by the evidence from Phase 1. The head teachers’ responses seemed to describe a different kind of effective partnership with APCs than is suggested by Programme Theory 2. Therefore Programme Theory 2 and the Programme Specification needed to be refined if they are to be viewed as describing the effective practice of the Treasure Project.
The researcher decided that it would be helpful to share her findings with the two APCs so that their own ‘folk theories’ (Tilley and Pawson, 1997) could be used to refine
Programme Theory 2 and add depth to the researcher’s understanding of Programme Theories 1, 3 and 4. The realistic interviewing of the APCs was seen as a first step for Phase 2’s deeper examination of effective partnership and effective Treasure Project CMO configurations. Following these interviews, the researcher planned to interview key
stakeholders in some of the most successful Project Schools so that more detail about effective CMO configurations could be revealed. Of particular interest to the researcher were the interviewees’ insights into the CMO configurations that enabled children’s mental health and emotional wellbeing to be promoted and/or resulted in enhanced capacity in schools to meet children’s emotional needs.
In order to increase the external validity of the process of choosing which schools were successful, the APCs were invited to be actively involved in the decision making process, given that they had insider knowledge about all 26 Project Schools. Therefore the
researcher designed the realistic interview to include time to select four schools each; two successful and two effective Project Schools. Although the APCs often worked together when training school staff or delivering projects, they each had responsibility for
developing a participatory partnership with a group of schools in a particular area, so it was likely that they would choose different schools. The researcher was aware that interviews would only be able to take place in the nine Project Schools where head teachers had indicated on the questionnaire that they were willing for both themselves and their school staff to be interviewed post-questionnaire. However, APCs were able to consider all 26 Project Schools before they made their choices. This was not felt to represent an ethical issue because the selection of the schools at this stage might or might not have led to their being asked to participate in interviews. By using this procedure it was hoped to reduce
threats to external validity that might have arisen through faulty sampling (Robson, 2002). The definition of ‘successful’ and ‘effective’ was left mainly unstructured, because the APCs might have differing constructs of these words and their interpretations may be related to their partnership relationship with the schools. The researcher thought it was more robust to allow them to pick schools which, in their personal view, were successful or effective, rather than asking them to match a school to the researcher’s view of what was a successful Project School, or effective partnership work. The APCs were able to pick the same schools for each category if they wished, in other words, a school could be successful and have an effective partnership relationship. The researcher also decided that thinking about the difference between ‘successful’ and ‘effective’ might strengthen the internal reliability of the choice of good sample schools. Because of the time constraints of the research, the researcher intended only to interview trained school staff and head teachers in two of the selected schools - one from each APC’s group of schools. The evaluation, as previously discussed, was concerned with revealing CMO configurations associated with successful outcomes, not unsuccessful outcomes. Therefore the researcher needed to use her time efficiently by closely examining CMOs in successful schools where the
participatory partnership had also been effective. She also decided that it was more balanced to conduct realistic interviews in one of each of the APC’s schools, so that any differences to do with personality, or way of working, were reduced. Therefore the researcher decided to conduct interviews in their most successful and effective school (if this school had also given permission to be contacted again for interviewing).