CONCLUSIONES Y RECOMENDACIONES

This evaluation was concerned with cause and effect since it sought to determine whether manipulating the intervention variable (training method) would cause changes in the outcome variable/s. In order to establish that any observed changes in the outcome variable/s were due to manipulation of the intervention (training method), any potential influence of other factors on the outcome variable/s needed to be minimised as much as possible (112, 115). Thus, the research design needed to be capable of employing methods that minimised any potential confounding of cause and effect through the control of extraneous variables (116). Additionally, in order to show association of cause and effect, a prospective design was needed to ensure that the manipulation of the intervention

variable (training method) preceded any changes in the outcome variable/s (112). Lastly, in order to prospectively measure any effect of intervention manipulation, the design needed to be longitudinal rather than cross-sectional (112). Thus, the optimal design to establish the effect of manipulating the intervention variable on the outcome variable/s was a prospective, longitudinal experiment. In particular, experimental designs employing randomisation and the use of a comparison group (randomised controlled trials) enable greater control of extraneous variables and improve accuracy in establishing possible cause and effect. The use of randomisation eliminates bias in the allocation of treatment,

maximises the likelihood of having similar group characteristics at baseline, and balances unknown as well as known confounding variables (112, 116). The use of a comparison group puts the effect of the intervention variable in context and provides a reference point from which comparisons can be drawn (117). However, the validity of any comparisons are dependent on the numbers enrolled and followed up in the experiment (112).

Randomised controlled trials (RCTs) can vary greatly depending on the design features that are employed (115). To assist researchers in matching the purpose of their evaluation to the ideal trial design, Thorpe et al (115) produced a multidimensional continuum to assess how explanatory or pragmatic ten key features of the evaluation are (the PRECIS tool). The authors provide a number of factors to consider for each domain to help researchers decide where to place that domain on the explanatory-pragmatic continuum. This process is illustrated for one of the domains, participant eligibility, in figure 12.

Figure 13. An illustration showing how a single domain (participant eligibility) is placed on the explanatory-pragmatic continuum

Since the evaluation of i-BeST was concerned with exploring efficacy, the ideal

experimental design was explanatory to ensure greater control of key variables. However, Thorpe et al (115) recognise that experiments are rarely purely ‘explanatory’ as it is often not possible to control all aspects of an evaluation. Figure 13 represents a retrospective graphical plot of the evaluation of i-BeST according to the PRECIS tool, where each of the ten key domains of the RCT have been rated according to how explanatory or pragmatic they were. The inner circle represents the most explanatory approach for each domain,

Explanatory hub:

 Use of selection criteria to restrict

individuals at risk of unfavourable outcomes  Criteria to maximise number of individuals

thought to respond favourably to the intervention

Pragmatic rim:

 Include all participants who have the condition of interest

 Eligibility is not affected by factors such as anticipated risk, responsiveness, comorbidities, and compliance.

Eligibility criteria

while the outer rim represents the most pragmatic approach to each domain. By way of comparison, figure 14 displays the original BeST trial (14) mapped according to the PRECIS tool. These illustrations are the interpretations of the thesis author (Helen Richmond) with regards to how explanatory or pragmatic the design features were in i-BeST and in the original BeST trial.

Figure 14. An illustration showing the how explanatory or pragmatic ten key design features were in the RCT of i-BeST

Follow-up intensity Outcomes Practitioner adherence Practitioner expertise (comparison) Flexibility of comparison intervention Practitioner expertise (experimental) Flexibility of experimental intervention Eligibility criteria Primary analysis Participant compliance

Figure 15. An illustration showing the how explanatory or pragmatic ten key design features were in the original BeST RCT (14)

Whilst the shape in figure 13 does not sit tightly around the inner circle on each of the ten domains, it clearly indicates that the experimental design was explanatory, with participant eligibility as the most pragmatic of the ten domains. In contrast, the illustration in figure 14 from the original BeST trial shows a much more pragmatic research design.

The use of a randomised controlled trial design is also supported in the literature by leading experts in the field of medical education (118). Here, the authors reviewed 110 studies of experimental design that investigated medical education interventions and proposed a framework to classify the designs. The framework used the term ‘justification studies’ to describe studies that made a comparison with another intervention to establish whether the intervention under investigation was better than, or as good as, the comparator. They noted the importance of these justification studies when evaluating higher order outcomes

Follow-up intensity Outcomes Practitioner adherence Practitioner expertise (comparison) Flexibility of comparison intervention Practitioner expertise (experimental) Flexibility of experimental intervention Eligibility criteria Primary analysis Participant compliance

(behavioural and patient- or practice-oriented outcomes). Since this thesis was concerned with the evaluation of two methods of implementing the BeST intervention, behavioural and practice orientated outcomes were considered. Therefore, according to Cook, Bordage and Schmidt’s (118) framework, a justification study would be the optimal method of evaluation. Additionally, latest guidance from the NHS pertaining to the implementation of technology enhanced learning (81) advocated that future research focus on the

effectiveness of online learning on clinical behaviours, thus further supporting the choice of study design described above.

4.5 Qualitative methodology