131
Chapter 6.
Evaluation of an assessment tool
Assessment of professionalism of dental students is important, both formatively, to aid professional development, and summatively, to provide assurance they have reached appropriate standards. The tools used for assessment need to be evaluated to ensure they are fit for the purpose.
Assessment needs to be based on explicit criteria. Chapter 4 defined professionalism in dentistry and operationalised this definition in a theoretical model, which was modified by changing the wording from “vocational” to “commitment” and from “altruism” to “consideration”, with “awareness” being replaced by “understanding”; for use on student assessment following panel testing (see Chapter 5 and Figure 6 page 86 compared with 8).
Figure 8 Model of Professionalism in dentistry adapted for use in assessment of senior dental students
132
This model was then used alongside a model of assessment in learning (Miller 1990) to produce a framework of educational aims and objectives within the existing curriculum. This framework was then used to produce an assessment programme (Appendix 1), within which is an tool, the Assessment of Dental Student’s Professionalism System (ADSPS), panel testing of which was reported in chapter 5.
The aim of this phase of the project was to evaluate this ADSPS assessment tool.
6.1. Method
Overview:
Test validation study using ADSPS in a cohort of senior clinical dental students within primary care settings.
6.1.1 Setting
Within the existing curriculum dental students in their fourth and fifth years of study attend three “Outreach” placements within primary care. These placements are either NHS general practices (6) or within the salaried dental services (5), one of which is a Direct Access Centre. Each placement receives between 2 and 5 students for six week placements. The placements take place throughout the second semester of the students’ fourth and first semester of their fifth year of study.
During the placement the students provide dental care commensurate with the nature of the placement. The general practices provide comprehensive dental care for
patients, many of whom are long term patients of the practice, within the regulations laid down by the NHS for general practice (Smith et al 2010). The salaried dental services treat children and patients with special care needs. The access centre cares for patients experiencing difficulty accessing routine care and the treatment carried out in this placement is often of an emergency nature (Smith et al 2010).
The placements are small establishments and students work under close supervision, provided continuously by the dental nurses and by dentally qualified clinicians. There is a very high ratio of dentists to students varying from 1:2 to 1:5.
133
The students manage a case mix similar to that seen in primary care which varies from placement to placement, but accumulates to a wide range of both patients and
procedures (Smith et al 2010). Students develop and demonstrate their abilities in a wide variety of contexts and Outreach was therefore considered a suitable part of the curriculum to pilot the ADSPS assessment tool. Furthermore, the close supervision meant that clinicians would be able carry out detailed assessment within a more holistic treatment setting and over a wider context than in the dental hospital clinics.
At the end of each placement students evaluate their experiences and specific student feedback on their experience is obtained. As each placement is evaluated at the end of each rotation, staff collect feedback and encourage reflection on students’ personal learning during each placement. These processes allowed qualitative evaluation of the assessment tool.
6.1.2 Participants
Recruitment of staff was initially by e-mail, followed by placement visit. Each placement was also visited and training and calibration was provided for the participating staff.
As there was no data upon which a power calculation could be based and for reasons of practicality, one complete cohort of students was requested to participate.
Recruitment of students was initially by e-mail followed by a lecture presentation. This sample was selected in the expectation that findings could be used to power a larger study at a later date, if this was deemed necessary.
The cohort consisted of 81 students of whom 44 were female.They were provided with details of the pilot during a specifically designed lecture at which they were encouraged to ask questions about the system. Each student and practice was also provided with a handbook explaining the ADSPS.
6.1.3 Assessment system
The Assessment of Dental Student Professionalism System (ADSPS) consisted of a meeting between each student and a supervising clinician at each Outreach placement, using three forms based on the dentists’ observations of the student.
134
Each student completed a self-assessment of their performance over the 6 weeks of the placement on a form containing 8 items relating to their professionalism (see Appendix 1b) based on an assessment. The supervising clinician completed an analogous form for each student (see Appendix 1b) which then formed the basis of a discussion about the student’s performance. This system was designed to encourage students to reflect on and explain their observed behaviours with appropriate
feedback. The appraisal was then recorded on a combined results form (agreed form), which had 16 items (Appendix 1b). A global rating of professionalism was collected separately for each student at the completion of placements. During the research the results were used to provide formative feedback only.
The forms were designed to record observed behaviour over time, whilst the appraisal interview allowed modification, where students could give an account of their behaviour. Thus, observed behaviours and the second order nature of professionalism were taken into account and the students were encouraged to develop their skills in both overt and tacit aspects of professionalism by reflecting on the staff feedback.
The global score and each domain were scored on a six point Likert scale where points 1 and 2 denoted students performing below the level expected from students at their stage of training. These scores related to the unsatisfactory level on the existing School system. The middle of the range (3, 4) represented students felt to be performing at the level expected (satisfactory). The upper range (5,6) represented students
performing well above the level expected (outstanding). Descriptors were provided for guidance when completing the forms (see Appendix 1b).
6.1.4 Permissions and liaison
Permission for the pilot was obtained from the University of Sheffield Ethics
Committee before participants were approached or recruited. Permission was also obtained from the teaching committee of the School of Clinical Dentistry, University of Sheffield to carry out the research within the school. Consent to participation in the research and to the meetings being recorded was obtained using a standardised consent form before any data were collected. For the students this was carried out as part of the explanatory lecture, whilst the researcher visited every placement and
135
obtained consent during the training process. Copies of consent forms can be found in Appendix 2.
6.1.5 Evaluation of ADSPS
The Assessment of Dental Student Professionalism System (ADSPS) was evaluated both quantitively and qualitatively.
6.1.5.1 Face and content validity
Face and content validity of the ADSPS had been evaluated by panel testing reported in chapter 5.
6.1.5.2 Response process
The response process describes the way an assessment system is used by the assessors and assessed, and was evaluated by descriptive analysis and by comparing the scores by students and staff over eight domains. The first available complete set of data was used for the analysis.
6.1.5.3 Reliability
6.1.5.3a Internal consistency
Internal reliability was evaluated via Cronbach’s alpha, alphas with each item deleted and corrected item-totals, on the agreed assessment form from the first available complete data set.
6.1.5.3b Test-retest reliability
Test-retest reliability was evaluated by comparing the first and second complete data sets. These scores were related to the student and not the tutor, as the student
136
placements were allocated randomly by the School administration. Intraclass correlation coefficient (ICC) was calculated for each item using a one-way repeated measures model. The global scores were also assessed as these were expected to show any changes in student development between placements.
6.1.5.4 Validity
6.1.5.4a Construct Validity
Construct validity was evaluated by assessing correlations between items within the model hypothesized to be related or unrelated. This analysis was based on the following hypotheses;
There would be a correlation between “self-awareness” and “reflection”
There would be a correlation between “self-awareness” and “vocational” aspects, especially self-motivation
There would be a correlation between “awareness of others” and “altruism” especially caring and respect
There would be a correlation between “trustworthiness” and “responsibility”
There would be a correlation between “ability to relate to context” and “accountability”
and the following hypothesis;
There would be no associations between the scores and age or gender
Correlations were expressed as these were evaluated using Spearman rho coefficients from the agreed assessment form from the first available complete set of data.
6.1.5.4b Criterion validity
Criterion validity was evaluated by correlating (Spearman’s rho) each item with the global rating using the agreed assessment form from the first available complete set of data. This global rating for each student was provided separately by the supervising clinicians at completion of the placements.