1. RESUMEN DEL PROSPECTO COMPLEMENTARIO
1.12. Resumen del Análisis Financiero
In this thesis we examine factors influencing observation and feedback in workplace- based assessment and learning. The research was conducted in the setting of postgraduate training in general practice in the Netherlands between 2009 and 2012. After a general introduction in which we pay attention to theoretical aspects of work- place-based learning and assessment, five chapters present the results of studies on elements of observation and feedback in workplace-based learning and the findings are synthesized and discussed in the final chapter.
Chapter 1 introduces the starting point of this thesis: an educational evaluation among general practice trainees in the Netherlands showing that numerous trainees are rarely if at all observed during single patient encounters. Similar findings have been reported in the literature on undergraduate and postgraduate medical education. The introductory chapter starts by painting a broad picture of workplace- based assessment and underlying theoretical perspectives, in particular Miller’s pyramid, formative and summative assessment, and the model of programmatic assessment of Van der Vleuten. Next the focus narrows to observation and feedback, and three elements are identified that are central to workplace-based learning and assessment: the trainee, who needs to develop; the trainer, who guides and assesses the learning process; and a real patient setting which cannot (and should not) be fully structured or standardized for assessment purposes but ensure fair assessment and useful feedback, probably using an assessment tool. The general objective of this thesis is to clarify observation and feedback in workplace-based assessment. To unravel this process we focus on assessment instruments, the influence of the trainee and the influence of the trainer.
Chapter 2 presents a review of the literature on instruments for assessing single pa- tient-trainee encounters in the workplace aimed at helping trainees gain insight into the strengths and weaknesses of their clinical performance. Most of the instruments reviewed were developed for formative assessment. Generally, they were given a positive reception, their feasibility was deemed to be good and acceptable and reliability was achievable with ten encounters. Of many instruments the validity was not investigated, but the validity of the mini-CEX and the clinical evaluation exercise was supported by strong and significant correlations with other valid assessment instruments. Evidence for educational effects was not very convincing. None of the studies included in the review examined effects of instruments on learning, clinical
er 8 Summ ar y
158
159
we conclude that it are the users rather than the instrument that determines the feedback process. That is why it seems advisable to direct interventions to improve the educational effects of feedback not at feedback instruments but at the users of those instruments.
Chapter 5 builds on the results of Chapter 4 by examining the relationship between reflection, feedback and the formulation of an action plan. Despite the absence of empirical evidence, the literature suggests that (self)-reflection can have a positive effect on the use of feedback. We hypothesized that feedback and reflection present a cumulative sequence and that reflection impacts on the use of feedback. In other words, trainers who provide specific feedback are more likely to pay attention to trainees’ reflections, and reflections in turn affect the use of feedback, with trainees who reflect on their performance being more likely to make use of the feedback they receive. We did this by examining the assessment forms used in Chapter 4. The results support both hypotheses. We found a cumulative sequence of feedback and reflection as well as an effect of reflection on the use of feedback. Specific reflections were found in combination with specific feedback only and they occurred more often in combination with specific action plans. This provides new evidence that (self)-reflec- tion can promote the use of feedback. Apparently, a reflective attitude towards one’s own performance plays an important role in applying feedback to enhance learning and improve performance. The feedback-reflection-action sequence might be used for faculty development in the area of feedback. Feedback has received considerable attention in (medical) education for quite some time, and trainers have learned a great deal about the importance of feedback and how it is best provided. The realization of the importance of reflection and its relationship with the use of feedback to set learning goals is a later development, however, and these notions have not yet found their way into the standard feedback procedures of many trainers. In Chapter 6 we examine the trainer as feedback-giver. The literature offers some insights into this role but a great deal remains to be discovered about factors influencing feedback-giving behaviour. We studied factors within the personality of the trainer, hypothesizing that self-efficacy, task perception, neuroticism, extraversion, agreeableness and conscientiousness influenced the frequency, quality and the consequential impact of feedback. We found that ‘task perception’ and the personality trait ‘neuroticism’ affected the frequency and quality of feedback. Trainers who saw it as their task to organize and support the use of feedback gave more and better feedback, while trainers with a more neurotic personality observed trainees more frequently and gave better feedback compared to trainers with an emotionally more stable personality. A possible explanation may be that the more neurotic trainers felt insecure about leaving patients in the hands of trainees and skills or the quality of patient care. There was substantial variation between instruments in
the competencies assessed, rating scales, frame of reference and user training. Assessment characteristics remained implicit and interpretation was largely left to the trainers. The effective implementation of instruments for workplace-based assessment probably hinges on appropriate training for those using them.
Chapter 3 focuses on formal feedback to trainees after an observed consultation. The assumption underlying the use of feedback is that it makes trainees aware of shortcomings and motivates them to improve their performance. Trainees were interviewed about external conditions affecting feedback after observed encounters in workplace-based assessment. The results revealed several conditions for enhancing the effects of feedback: 1) observation and feedback are planned by the trainer and the trainee; 2) the content and delivery of the feedback are adequate; and 3) the trainee uses the feedback to set learning goals to guide learning. All trainees showed primarily emotional responses to observation, notably apprehension, but differed in the strategies they used to deal with it. While some avoided observation, others overcame their apprehension and actively sought observation and feedback. Avoidant trainee behaviour could be counteracted by an active trainer. Trainers also fell into two groups: those actively initiating and those refraining from observation and feedback.
The study in Chapter 4 investigates written feedback after observed patient encounters entered in a form that was designed to meet the following general notions about feedback. Feedback is preferably narrative and specific, explicating where more work needs to be done. Feedback can be made more effective when recipients receive guidance on turning feedback into steps to improve performance. Feedback is preferably a two-way process with trainers providing comments and at the same time encouraging trainees to self-reflect on their performance. We studied the effects of an assessment instrument designed to stimulate trainers to provide written feedback, trainees to write self-reflections and trainer and trainee to write an action plan derived from the feedback and reflections. Trainers provided feedback more frequently than trainees wrote self-reflections, while action plans were very rare. The written comments were generally specific. Because in the context of this study a trainer and a trainee team up for the duration of one year and because trainer- trainee teams showed huge differences in the number of forms handed in, we examined the specificity of comments using trainer-trainee teams as the unit of analysis. Differences proved to be large, with some teams consistently writing specific comments, while others consistently failed to do so. Although the layout of the form indicated which comments were to be entered in which spaces, in the end it was up to the users how to use the form. Based on the substantial differences between trainer-trainee teams
er 8 Sam en vat ti ng