Introducción

2.6.1 Summary of findings

The evidence base (RCTs) for estimating the effectiveness of online learning in this

population is limited. Whilst eleven RCTs and one cluster RCT were included in this review, many of the sample sizes were small. All included studies were of poor methodological quality. For the majority of studies, the interventions and study results were poorly reported, with data to calculate effect sizes available in only six studies. Due to

methodological, statistical and clinical heterogeneity of the included studies, it was not appropriate to pool these effect sizes in meta-analyses. The effect sizes showed no statistically significant differences between online learning interventions and more traditional, face to face or paper-based interventions on health care professionals’

knowledge, practical skills or clinical behaviour. There were no clear directions of effect for any of the outcome measures. Whilst this could be a result of the previously mentioned limitations of the included studies, such as a lack of power, it could also be interpreted to mean there were no actual differences in effectiveness between online learning and traditional methods (such as face-to-face). This may suggest that online learning is a viable alternative to traditional methods of training.

It was also not possible to isolate the effect of specific intervention components on any of the outcome measures. Thus, this review was not able to provide any information about which components of online interventions may result in greater satisfaction and

effectiveness of learning for health care professionals. None of the studies used validated outcome measures or any framework for evaluation of the online learning programmes. Additionally, since none of the studies addressed level four outcome measures (all were concerned with levels 1-3: health care professionals’ reactions; their learning; and any transfer of knowledge), the review cannot provide any information as to whether online

training can produce equal or superior results for patients compared to training in traditional formats.

2.6.2 Issues concerning the variations in observed effect sizes

Firstly, many of the included studies were small with half of the RCTs including less than 50 participants and only two studies recruiting more than 100 (67, 68). Six of the ten RCTs used more than two comparison groups and therefore needed a greater number of participants to detect any intervention effects. In addition to the small sample sizes, the wide confidence intervals around the effect sizes are an indication that the studies were under powered (76). Secondly, the differences in study control groups may have

contributed to the varying direction of effect sizes. In two comparisons (52, 75), the intervention was compared solely to a paper-based control. Effect sizes could only be calculated for the first study, though study results from both papers suggested effects in favour of the intervention groups. On the other hand, there were four comparisons of the intervention to interactive, small group teaching (52, 66, 70, 73) where all effect sizes favoured the control groups. However, none of these effect sizes were statistically significant and therefore no conclusions can be drawn between the effect sizes and the different control groups.

Thirdly, the fidelity of the interventions themselves should be considered. One of the cited advantages of online learning is the flexibility of learning, enabling users to learn in an environment and time suitable to them (49). However, few studies actually reported participant log-in times, and even amongst those that did, these times do not necessarily reflect active learning. Contrastingly, in a small interactive group, the researchers were able to ascertain the time the participants were engaged in active learning. Thus, without recording the adherence of participants in the intervention groups, it is not possible to ascertain whether the learning was carried out as intended, which in turn may have resulted in effect sizes favouring the control group. Another factor to consider is the

varying lengths of the interventions and their controls. The length of the intervention and control were only reported in 7 of the 11 studies and in 50% of these, the duration of the intervention differed greatly from the duration of the control. Thus producing another potential confounding variable when trying to determine the effectiveness of online learning.

Additionally, the components/content of the interventions, which were poorly reported in the majority of studies, should be considered. Indirect effects such as costs, how the online learning programme was developed and who developed it could impact upon the

effectiveness of the intervention (77, 78). For example, a specialist company may produce a programme of greater technical quality than a local health care professional involved in research. As previously mentioned, specific design features, such as interactivity, are associated with improved learner engagement and satisfaction (53). Thus, engaging online programmes may have exerted greater effects than simple programmes. Unfortunately, poor reporting of both the study interventions and results meant that the effects of these components could not be isolated; thus, any relationship between particular design features and effect sizes could not be explored. This poor reporting also precludes future use of the interventions in clinical practice or in future research, since the authors do not provide enough information to reproduce the interventions (79).

Lastly, several of the outcome comparisons contained studies with multiple control groups. Therefore, multiple comparisons against the same intervention, with different control groups, were included. Since the effect sizes were not pooled in meta-analyses this is not of great concern, however, the inclusion of multiple comparisons from a single study for the same outcome mean that any biases of that study will have occurred twice within that outcome (60).

2.6.3 Current findings in relation to BeST

BeST is a complex intervention and as such, the process of training health care professionals to effectively deliver it as intended, is also complex. Two studies in this review investigated the use of online learning to train clinicians in a CBT-based intervention (52, 68). One of which was a brief module that aimed to raise knowledge on a topic with a narrow scope (68), while the other trained clinicians in a fully manaualised treatment intervention (52). The authors of this latter study raised the issue of a lack of evidence evaluating online learning for delivering training in manual guided psychotherapies. This is supported by this review, where the majority of studies investigated competency of a single practical skill or gains in factual knowledge. Whilst this review identified one study of a similar complexity to the BeST intervention (52), the study itself had many methodology flaws. For example, the development of the intervention was not described, and their randomisation process failed, with over half of the participants allocated by choice (52). Thus, the study does provide some basis for investigating the effectiveness of online learning in training health care professionals to deliver a structured CBT intervention; however, it does not provide conclusive or substantive evidence to draw on.

2.6.4 Current findings in relation to the literature

Despite the stricter and more precise inclusion criteria of the current review, the findings are in agreement with those of Cook et al (49), who conducted a broad systematic review on the effectiveness of online training and found a lack of effect size in either direction when online training was compared to alternative forms of training. Since the effect of specific intervention components could not be isolated in this review, the findings cannot be compared to the later review by Cook et al (53), which investigated design variations of Internet based learning.