A mixed methods design was used in this research. The quantitative data were observational records of the children's progress over three time points. All the interviews provided qualitative data. Parental, LEAP specialist and preschool staff views were obtained through interviews at the start, during (with the exception of preschool staff) and at the end of LEAP to provide more insight than would be possible from just one point in time. Initial interviews with preschool practitioners took place in April rather than
immediately after the first parental interviews in March because it was more convenient for preschools due to Easter preparations and holidays. The timing of these interviews meant that no time two interviews took place at the
52
preschools because a subsequent interview in May would have been too soon after the initial interviews. Obtaining views at three time points has been recognised as more effective for gaining information about how views change over time (Brooks, 2016), and Ployhart and Vandenberg (2010) have remarked that when only two time points are used it is difficult to determine the course of any change that has taken place. The use of three time points for observations and interviews therefore provided more insight into whether any changes were noticed early in the intervention or later.
A mixed methods approach was considered appropriate because it would mean that complementarities could be obtained (Plano Clark & Ivankova, 2016). Gaining different perspectives in this way would mean there would be less reductionist understanding than if a single method was used (Mertens, 2010). In addition, Creswell (2009) has noted that mixed methods are suitable when using either quantitative or qualitative alone would be insufficient.
Conceptual issues including how those involved felt about the evaluation needed to be anticipated when the evaluation was devised (Patton, 1987). Therefore, political factors which related to the LEAP evaluation needed to be considered. The EP who developed the programme was obviously keen to know whether LEAP was beneficial but recognised that as it was a pilot programme, changes would be inevitable. This view meant that potentially some concerns about LEAP raised by participants before or during the programme would have the opportunity to be resolved providing
53
was important to consider how to conduct the research ethically, feasibly and skilfully while maintaining its usefulness for the stakeholders (Robson & McCartan, 2016).
A qualitative evaluation was useful to explore individuals’ varied experiences of LEAP and capture an array of individualised outcomes (Patton, 1987). An evaluation using a traditional experimental evaluation would only have provided descriptions of the outcomes rather than provided explanations of why a programme might work or not (Pawson & Tilley, 1997). Greene, Caracelli and Graham (1989) proposed a conceptual framework relating to the design of mixed method evaluations presenting their different purposes. They noted that triangulation would seek corroboration of results from
different methods while complementarity would seek elaboration. Both would increase the validity of constructs although complementarity would provide more flexibility of design and would be used, as in the case of evaluating LEAP, to assess aspects of a LEAP in addition to changes observed (Greene, Caracelli & Graham, 1989).
Pawson and Tilley (1997) note that the design of an evaluation should enable identification of subgroups where the programme has been more or less successful as this would help to provide insight into the possible reasons for the programme’s success. It would be beneficial for this design to enable subgroups in LEAP to be considered. However, it would be important to be cautious with these interpretations about possible success or failure as the study is not experimental.
54
There was no control group in the study indicating that the design is quasi- experimental (Shadish, Cook & Campbell, 2001). There are limitations with this, such as alternative explanations for outcomes which should be specified in advance. However, it is difficult to consider all alternatives prior to the research as not all are predictable (Shadish, Cook & Campbell, 2001).
The type of intervention under investigation adopted techniques to best fit the individual family's and child's needs, and this was difficult to investigate empirically with a control group because all the children's needs were different. There was also no control group because of ethical concerns. There was no waiting list for children to commence LEAP and it would be inappropriate to expect children (and their parents) to experience the repeated observations both at home and in preschool settings if they were not involved in the intervention. A comparison group such as Portage was considered for the current study and has been used in prior research investigating LA services (Reed, Osborne, Makrygianni, Waddington, Etherington & Gainsborough, 2013). However, the Portage service in the current researcher's LA ends when children are three years old and so a group of children receiving Portage were not considered comparable in terms of chronological age.
However, without a control group the observation data alone cannot be used to establish the effectiveness of LEAP. Any changes that occur over time could be due to natural development rather than to the intervention (Shadish, Cook & Campbell, 2001). An additional method of investigating
55
practitioners' views about whether the intervention had an impact. Using mixed methods in this way would also mean that possible reasons for any changes could be investigated. As Maxwell (2004) noted, involving different methods and individuals from different settings in investigations reduces biases occurring through use of limited settings or employing just one method.
SCERTS Assessment Process Observations (SAP-O) of the children were used because these are discreet methods for obtaining data relating to the children in different situations and settings (Prizant et al., 2006). The LEAP specialists were trained in using the SCERTS model and part of their role involved completing the SAP-O forms to record children's progress. Each SAP-O form was completed by two members of LEAP staff to improve the reliability of the scores recorded. The current researcher and LEAP
specialists would not have had time to use any additional observational tools to record progress and such use would have been intrusive to children, families and the preschools. SCERTS observation scores for the children were not accessed until all time three interviews were conducted in order to avoid interviewer bias.
Semi-structured interviews were used to obtain views from adults involved in LEAP because they enable the interviewee to talk readily, points made can be clarified and comments can be probed further (Howitt & Cramer, 2007). Arguably an unstructured interview could encourage interviewees to focus on their main views. A semi-structured interview was preferable in this case because the discussion needed to centre on issues relating to the research
56
questions (Brinkmann, 2014). One researcher was involved in interviewing the participants, which enabled a consistent approach to be maintained across all interviews. Having one researcher meant that it was easier to refer to comments made by interviewees in earlier interviews and also meant that a rapport could be built up over time that was sensitive to the
researcher-interviewee relationship (Galletta & Cross, 2013).
As a less experienced interviewer at the outset of this research it was more suitable to use complete questions rather than solely themes or topics in the interview schedule so that there was no need to devise the format of the questions during the interview (Horrocks & King, 2010). Horrocks and King (2010) also recommended including probes and prompts. Probes were included to enable further details about participants' responses. Prompts were added to the schedule to assist participants when they were uncertain about their response to a question.