Simulación de la labranza

Since this study aimed to conduct a pragmatic exploration of two implementation strategies, a range of outcome measures were selected to evaluate both the comparable effectiveness of i-BeST for training physiotherapists in the BeST intervention, and to

compare the effectiveness of both implementation strategies as a whole. A widely

accepted approach to the evaluation of training, as previously discussed in chapter two, is the four-level model proposed by Kirkpatrick (61). Since its unveiling over 50 years ago, the model has been updated to expand the definitions of each of the four levels (140), shown in Figure 17. The levels progress sequentially, with level one measuring the simplest outcomes from a training programme, participants’ reactions, such as their satisfaction or engagement with the training. Level two refers to participants learning - how much they have acquired the desired knowledge, skills, attitudes and confidence. Becoming more complex, level three evaluates the transfer of knowledge, skills or attitudes into practice, thus measuring behaviour change. Kirkpatrick noted that this level is usually measured through observation and at three to six months following the training (137). Lastly, level four refers to the desired results of the training, in this instance, patient related outcomes.

Kirkpatrick advocated that evaluation of training programmes should measure outcomes at each level of the model sequentially with the ultimate aim of reaching level four, deemed to be the most advantageous level of evaluation (137). However, as Cook and West (141) caution, the link between therapist behaviour and patient outcomes is not direct, with

Results Evaluation of behaviour Evaluation of learning Evaluation of reactions

Impact of training on patient outcomes Transfer of training leading to changes

in clinical practice

Acquisition of intended knowledge, skills, attitudes and confidence

Degree of satisfaction or engagement with the training

multiple confounding variables at play, potentially diluting training effects. Thus, evaluation of a training programme based on patient outcome measures would require very large samples to detect the diluted training effects (138). Additionally, Cook and West (141) highlight that much can be learnt from outcomes in line with levels one to three, suggesting that educators need to first ensure that participants behaviour can be influenced by a training programme, before progressing further to assess patient effects.

Since this was a small exploratory study with a restricted time frame, it was not appropriate, nor desirable, to assess patient outcome measures. Therefore, outcome measures were selected in line with levels 1-3 of Kirkpatrick’s four-level training evaluation model (62). With reference to level three, it was important to assess the actual behaviour of participants in relation to both the number of participants implementing the BeST intervention and their clinical competency in doing so (detailed below). When considering level two evaluation measures, it was important to assess change (if any) in the attitudes and beliefs of participants, their self-efficacy to perform the desired skills, and their knowledge of the BeST intervention (detailed below). Participants’ engagement and satisfaction with the training, level one measures, were also assessed to ensure that outcomes at all three levels were included.

Kirkpatrick’s evaluation model: level three outcome measures

Implementation of the BeST intervention

The number of participants setting up and delivering the BeST intervention was recorded. This enabled comparison of baseline and outcome variables between those delivering the groups and those not.

Clinical competency in the BeST intervention

Competency was measured using the Cognitive Therapy Scale – Revised – Pain tool (CTS-R- Pain; appendix 12). This tool has been specifically modified to measure competency in the use of a CB approach among non-psychology specialists (142). Hansen (139) found the tool

to have high internal consistency (Cronbachs α = 0.99) and good inter and intra-rater reliability (intra-class correlation coefficient for intra-rater reliability = 0.92 (0.79; 0.97) and inter-rater reliability amongst 4 raters = 0.82 (0.30; 0.99)).

Once a participant had arranged the dates for their BeST group sessions, a randomly selected group treatment session (1-6) was audio taped and evaluated using the CTS-R-Pain scale. A novice rater (Helen Richmond) and an experienced rater (Zara Hansen) both independently scored the first three audio recordings, resolving any disparities through discussion, to ensure consistency. The novice rater then assessed all further recordings.

Scoring

The tool consists of 15 items, each assessing a key competency of the CB approach.

Participants are scored on a scale of 0-6 for each item. Narrative descriptions and examples of each level are provided to help guide the rater.

Interpretation

A higher score indicates greater competency, ranging from a score of zero, categorised as ‘incompetent’, to a score of six, categorised as ‘expert’ (Figure 18). All 15 items were not necessarily scored, for example, item 15 was only scored if it occurred during the session. Therefore, the total score for each participant was constructed from the mean of all scored items.

Figure 19. Categorisation of CTS-R-Pain competency scores

Competency level Examples

0 Absence of feature or highly inappropriate

performance

1 Inappropriate performance with major

concerns evident

2 Evidence of competence, but numerous

problems and lack of consistency

3 Competent, but some problems and/or

inconsistencies

4 Good features, but minor problems and/or

inconsistencies.

5 Very good features, minimal problems

and/or inconsistencies

6 Excellent performance, or very good even the

face of patient difficulties

Kirkpatrick’s evaluation model: level two outcome measures

Attitudes and beliefs towards the management of chronic LBP patients

Mutsaersa et al (143) describe attitudes as underlying properties that effect behaviour, synthesised from multiple beliefs. Evidence suggests that the attitudes and beliefs held by practitioners can influence both their treatment recommendations and their perceptions of their patients (144, 145). Thus, a key aim of the BeST training programme was to shift the attitudes and beliefs of participants away from the traditional biomedical model of health, towards a more psychosocial model of health. Therefore, it was important to measure the attitudes and beliefs of participants regarding the management of chronic LBP patients before and after their training.

Two widely used tools for assessing attitudes and beliefs are the Health Care Provider’s Pain and Relationship Scale (HC-PAIRS (145)) and the Pain Attitudes and Beliefs Scale for Physiotherapists (PABS-PT (144, 146)). Since its initial development, the 20-item PABS-PT

Expert Proficient Competent Advanced beginner Novice Incompetent

scale was further validated to a 19-item tool and tested in multiple populations and contexts, showing good reliability and validity (143). In addition to its wider use, and in contrast to the HC-PAIRS, the PABS-PT scale measures psychosocial (9 items) as well as biomedical factors (10 items). A recent systematic review (143) found the PABS-PT scale to be responsive to the educational intervention being tested in all of its included studies.

Therefore, the PABS-PT scale (appendix 13) was used prior to and following the training to measure participants’ biomedical and psychosocial attitudes regarding the management of chronic LBP.

Scoring

Participants scored 19 items (statements)on a six-point Likert scale ranging from totally disagree (score 1) to totally agree (score 6).

Interpretation

Responses to 10 of these items are collated to give a score for the biomedical factor (factor one), which has a range of 14 to 84. A lower score indicates that the participant holds less biomedical attitudes and beliefs towards the management of chronic LBP. To calculate the psychosocial factor, responses to 9 of the items are collated, which has a range of 6-36. Again, a lower score indicates that the participant holds less psychosocial attitudes and beliefs to the management of chronic LBP. Therefore, following the training, the desired response was to see a decrease in the participant’s biomedical score (factor one), and an increase in the participant’s psychosocial score (factor two).

Knowledge

Adequate knowledge of the BeST intervention, including its underlying rationale and principles, was essential for the individual to both adopt and deliver it (32).

Participants’ knowledge of the BeST intervention was assessed following the training using a self-developed multiple-choice questionnaire (appendix 14). Face and content validity of

the questionnaire was enhanced by devising the questions from the BeST training material and matching them to each aim of the training programme (26). The questionnaire was also piloted with a content expert (Dr Zara Hansen).

Scoring

Participants gave either a single or multiple responses as instructed per question.

Interpretation

The maximum score available was 31. One point was awarded for each correct score. Where multiple responses were asked for, one point was awarded for each correct response, and half a point was deducted for each incorrect response.

Self-efficacy

Self-efficacy is a belief in one’s ability to perform a given task in a specific context and constitutes a significant parameter in behaviour change theories (147). Individuals reporting high self-efficacy are more likely to adopt and commit to delivering an

intervention regardless of the obstacles they are presented with (32) , suggesting that self- efficacy plays an important role when aiming to achieve changes in an individual’s

behaviour. Therefore, participant self-efficacy to implement the BeST intervention assessment and group sessions into their clinical practice was evaluated.

Bandura (148) warns against the use of an ‘all purpose’ measure of self-efficacy, stating that these generic scales have limited predictive ability and, since they are detached from any given context, have little informative value. Thus, self-efficacy scales need to be tailored to the particular behaviour that is being studied. Since this study wanted to evaluate participants’ self-efficacy to deliver a specific intervention, a measure was developed that was specific to the BeST intervention that included two 10-point likert scales. Attention was paid to the wording of the scale, ensuring the use of ‘can do’ to signify capability (145).

Scoring

Participants marked a line with a cross to indicate how confident they were to firstly, conduct a BeST patient assessment, and secondly, to deliver a BeST group session.

Interpretation

On a scale of 0-10, a higher score equated to greater confidence. Satisfaction

Poor compliance and non-completion are recognised obstacles to successful online learning (149). Learner satisfaction has been shown to correlate with user engagement, intention to use, and compliance with the training programme (150). Additionally, Sun et al (77)

stipulate that participant satisfaction is a key outcome for establishing the success of implementation. Therefore, measuring participant satisfaction was important to ascertain any association with engagement and compliance, and to explore acceptability of the training methods.

Satisfaction with training was assessed using a self-developed questionnaire asking participants to record their level of satisfaction, ranging from very unsatisfied to very satisfied. Free text boxes provided space to detail likes, dislikes and suggested improvements (appendix 16).

Learner analytics

Learner analytics were recorded for participants in the i-BeST training arm to establish their engagement with the online programme (detailed in chapter 6a). This included detailed logs of online behaviour such as duration of log-ins, slides accessed, length of time per slide and materials accessed/downloaded.