2.2 Marco Conceptual
2.2.2. Inclusión de estudiantes con necesidades educativas especiales
In Chapter 2 (Section 2.7.2) the relationship between the NPIP WBA (as understood
then) and the strengths and limitations described in the WBA literature were explored.
This was examined in terms of the competency framework, rating scale, the one-to-
one relationship between the interns and tutors, intern self-assessment, tutor
assessment, meetings to discuss assessment, and role of Compass technology. I
focus groups in this chapter. The role of technology is considered separately in
Section 4.5.3 below.
In Chapter 2, before completing this empirical work, it was not possible to identify if
the CCF represented a strength due to its national focus (Holmboe et al., 2010), or if
this made it too general to reflect the usual activities of the workplace (Lurie et al.,
2011, p. 49). The focus group findings identified that in practice, the CCF was
reported to have both strengths and some limitations rather than being either ‘good’ or ‘bad’. Many participants indicated that the CCF provided a structure that was helpful in informing them what they needed to achieve. This reflects findings from a recent
interview-based study by Storrar, Hope, and Cameron (2018) which found that
trainees appreciated clear guidance on what constitutes competence and therefore
what they needed to achieve to progress. In other studies, trainees have reported
understanding frameworks to be challenging due to terminology (Lomis et al., 2017),
but the participants in the focus groups did not report this as a challenge. The tutors
agreed that the CCF structure helped them approach the WBA in a more balanced
manner, and highlight positive aspects of the intern’s behaviour, rather than only “looking for alarm bells for things that are going wrong”. Some participants,
particularly in non-clinical placements, found some elements of the CCF not relevant
to their context. Affected interns found this disheartening (e.g. “I found it very
difficult to do because there was huge chunks that are not applicable, so you’re there
scrolling down going, can’t do that, can’t do that’, and that’s a little bit disheartening
as well, you looking at a whole big list of things going, I have no idea what any of this
is about”). Some tutors sought to overcome this tension by developing local solutions
(e.g. “Well to be honest, out of the 144 (sic) I think there’s just so many. We did an
spreadsheet of all the core competencies and we matched the behaviours described in
the core competencies to what the actual day job of the intern was within our section”)
(labelled xvi, Fig. 4.7). Adapting national frameworks to local teaching contexts is
known to be challenging (Delany et al., 2016), and this description of the tutor’s
efforts to overcome this represents an example of expansive learning (labelled xiv,
Fig. 4.7). Interestingly, the tutor’s approach described appears to have similarities to
the EPA-based approaches now commonly employed to overcome the limitations of
checklist-based WBA. With EPAs, competencies (which are considered to represent
personal qualities) are reformulated into units of work that can be observed as part of
standard workplace activities (Caverzagie et al., 2015; ten Cate & Scheele, 2007).
The next consideration is the rating scale, which was identified as both a tool for
achieving the object and a rule that mediates how those involved work together. The
interns generally reported finding the scale helpful as a tool for tracking progress over
time. There were some indications that it was challenging to use (e.g. “So there is a lot
to think about at once…”) but in general the interns did not report many issues with
using the scale. Interns indicated that they felt that seeking to achieve a Level 4 rating
as competent did not facilitate development in areas beyond minimal competence (e.g. “It’s just kind of like, this person is going to give me a fours anyway…so…it’s kind of like maybe there are other things I could have improved on…but you don’t
know.”). This represented a secondary contradiction between the assessment as being
a requirement but also playing an important role in development (labelled ii, Fig. 4.7),
and reflects concerns in the literature over WBA and reductionism. The tutors
similarly expressed concern that the rating scale as a rule did not facilitate recognition
of excellence, as the requirement was for sign-off at level 4 or competent, (for
the CCF, it just doesn’t make sense. They agreed that having a scale that could reflect
natural variance in ability beyond the level of competent would be useful in more
accurately assessing their intern’s progress, providing balanced feedback, and
encouraging excellence rather than minimum competence (labelled xiii, Fig. 4.8).
They appeared to feel that the WBA was simultaneously ‘tick-box’ and useful, which
reflects previous findings from a survey-based study of academics (Dobbins, Brooks,
Scott, Rawlinson, & Norman, 2016). All interns and tutors referred to competence in
numerical form (e.g. (tutor) “you have to give between a one and a four”) which they
appeared to interpret similarly (e.g. an intern summarised “it’s like four for excellent,
and one for oh my God I’m useless”). This indicated some limitations in the use of the
CoDEG scale. Rather than referring to some kind of criterion that could meaningfully
guide development (e.g. I needed help to demonstrate X) which is the preferred
approach in WBA, they used language more traditionally associated with norm-based
assessment (Pereira et al., 2018). Construct-aligned scales e.g. the Zwisch (George et
al., 2014) and other scales (Crossley et al., 2011) may improve this process (Rekman,
Gofton, Dudek, Gofton, & Hamstra, 2016). However, the rich descriptions provided
by the interns and tutors indicated that while the assessment may ultimately be
represented as a number from the rating scale, each rating is based on considerable
thought. Capturing this in narrative form using a mixed-methods approach may help
improve assessment and feedback (Hoang & Lau, 2018).
The NPIP WBA requires a one-to-one intern to tutor relationship where they work
together a minimum of three days per week and the tutor is responsible for the intern’s
final rating at Level 4. The WBA literature suggests that this approach is not ideal,
and that multiple assessors should be used in determining the competence of a trainee
other hand, having too many assessors and rotations can impact on the trainees’ ability
to build relationships (Storrar et al., 2018) and may compromise patient safety
(Englander & Carraccio, 2018). The document analysis indicated that the intern and
tutor work alongside each other (community) and that they complete the assessments
as per the requirements (division of labour). However, the focus groups indicated that
while this is true, interns and tutors felt there were many others involved in the WBA,
whether their roles were formally recognised or not. This appeared to vary between
placements according to the interns (e.g. an intern who had undertaken placements in
different environments commented “…so it depended on each person’s individual
setting, how many people fed into your own assessment of yourself and your tutor’s
assessment of you”). While beneficial for assessment, this one-to-one relationship
resulted in contradictions arising from the perceived requirement to keep the tutor
happy; with interns agreeing that their behaviour was therefore influenced when their
tutor was present (e.g. “I ended up having to go with my tutor just for the
repercussions of it”). Tutors described how they included opinions of others in their
assessment of intern’s progress, including views of technicians, patients, and other
team members, although this did not reflect normative practice according to the
document analysis findings. As the tutor still made the final decision based on this
information, there was no contradiction evident, but it was interesting to note that it
was happening as part of practice to overcome perceived limitations of the existing
system. Multisource feedback is an established approach in WBA, and it appeared that
some tutors were intuitively identifying the recognised benefits of multiple assessors
(Lockyer, 2013). The fact that the relationship between the intern and tutor was one-
to-one, led to the manifestation of other contradictions, where other staff did not
(labelled ix, Fig. 4.7) or frustrated at the perceived superior learning opportunities
afforded to them (labelled x, Fig. 4.7). The clarification of trainees’ roles is an
important part of identity formation during workplace-based learning (Jarvis-Selinger
et al., 2012), and clarification was considered key to tutors who suggested developing
methods to clarify with other staff (e.g. information sheets) which represented a form
of expansive learning to overcome the identified problem.
The core of the WBA consists of intern self-assessment, tutor assessment, and
subsequent meeting to discuss the ratings and feedback. Most of the interns reported
finding this process useful for their development, and received feedback and
encouragement (e.g. “I would say it was constructive and I knew what to do and what
I had to improve, and even things I would have, again, marked myself lower and she
would have said ‘Why are you putting yourself that?’ So I think from that point of
view they were constructive.”). Others suggested that it became a tick-box exercise
(e.g. “Are they good enough? Are they okay? Yes, no?”). This represents a primary
contradiction within the rule (labelled i, Fig. 4.7). In terms of self-assessment, the
interns relied on using the CCF to stimulate thinking around examples of practice or
estimation of confidence. Repeated reference to ‘confidence’ rather than competence
appeared to suggest limitations in some interns’ self-assessment approach. Students’
self-assessments have been shown to vary from assessors’ assessments, with
interpretation of the framework and scale considered areas where variability arise
(Tallentire, Smith, Wylde, & Cameron, 2011). The activity system as informed by the
document analysis and focus group data highlights the training tutors receive as being
an important tool, but findings indicate that the training may oversimplify the
complexity of WBA to a degree. This aligns well with the literature on WBA which
Lockyer, Carraccio, et al., 2017, p. 612). Rather than mentioning being trained on
rating scales, tutors reported applying skills learned in training during WBA
(“especially if you are giving negative feedback…and how to give it in the best way”).
One of the main problems manifesting with the WBA was having sufficient time to
complete the appraisals, particularly for the first one, which took the longest. Interns
and tutors both described conflicts between WBA and the role of the intern and tutor
as delivering professional services and completing academic programmes in parallel.
As the WBA was not completed using assessment tools designed for real-time
observations (e.g. mini-CEX), it required a separate meeting, which was a challenge
for some leading to a quaternary contradiction between the WBA and the general
activities of the workplace (labelled xiv, Fig. 4.7). This was described many
participants in both groups (e.g. one tutor stated, “You have ideas and plans and set
time aside then you get an urgent query from ICU. The time management aspect of it
is very difficult.”). The balance between the role of trainees as learners and assessors
as service providers is an established challenge for WBA (Nousiainen et al., 2017, p.
596). The WBA is completed as two formative assessments, and one summative, and
while there are no target scores for the first two assessments, tutors indicated that they
expected to see development over time, and some looked for certain general
indicators. One tutor stated that they expected to see development across the three
WBAs as follows, “I always…[expect that my] intern in community [pharmacy]
[progresses from] an excellent OTC assistant, [to] an excellent technician and [is
finally] an excellent pharmacist”. A developmental trajectory is formally required in
many medical CBME programmes, where the trainees should develop in accordance
with particular ‘milestones’ (Lowry, Vansaghi, Rigler, & Stites, 2013), but this is not
describing learning, but instead focused on more holistic decisions. This aligns with
the findings of Regehr et al. (2012) who studied faculty decision-making and
demonstrated that faculty can make consistent decisions without relying on
deconstructed competencies, although this does not reflect the aims of faculty
development provided to try to achieve ‘reliable’ ratings (Pelgrim et al., 2011).
Feedback is considered an important aspect of WBA and contributes to trainee
acceptance of WBA (Ross et al., 2012). Interns considered feedback an important tool
for learning, although not all interns reported receiving it, and such variability has
been reported in other WBA studies (Holmboe, 2004, 2015). One intern suggested
that having textboxes after each behaviour to facilitate gathering of information could
be useful where tutors were not making the time to provide feedback as “at least that
would prove maybe that they’re doing something”. As well as introducing a level of
proof of engagement, this would also facilitate recording of more narrative comments,
which is useful for gathering holistic information for decision-making (Ginsburg et
al., 2011). It could also add credibility to the feedback and address emotional
obstacles if given in the context of a conversation (Tekian et al., 2017) . Feedback is a
vital component of WBA, and it was concerning that this important element of WBA
was not being provided to interns in some cases, although this variability has
commonly been experienced elsewhere (Holmboe, 2004, 2015).