Interviewers
Demographic dataOut of 41 interviewers, 36 (88%) of them completed the questionnaire. All the interviewers were at the consultant level with at least one year‟s experience at that grade. The mean age of the interviewers was 46 years (standard deviation (SD) 7.2; 95%Confidence Interval (CI) 43.5, 47.5) and 56% of the interviewers were male. The vast majority of the interviewers (81%) had time in their job plan for education. Professional leave was taken by 80% of the interviewers to attend the two interview days and 89% of those who took professional leave had no problem with obtaining leave for the trainings and interview days, however, 5.8% of the interviewers had used annual leave to attend the interview.
Responses to the statements
Out of the 20 questions in the questionnaires, more than 90% of the interviewers strongly agreed with the following eight statements:
„I would consistently offer the same assessment if asked to review my opinion at a later date‟,
„Interview stations were of the right duration‟,
„Interviewers usually agree when comparing their assessment of candidates at different stations‟,
„Our interviews appeared fair to the candidates‟(Figure 3),
„The interview format selects the best candidate to be appointed‟,
„The same panel conducting the interviews for a second time, several weeks later and with the same candidates, would appoint the same people‟,
57 | P a g e
„We asked searching questions during the interview‟and
„We tested an appropriate range of competencies during the interview‟. More than 75%, but less than 90% of the interviewers agreed that:
„Performance at interview predicts future performance‟ and
„The format was better than that of a traditional panel‟. (Figure 5). More than 25%, but less than 50% of the interviewers mentioned that:
• „The candidates‟ personality influenced my assessments‟,
• „The interview helped candidates to understand what a good orthodontist is‟ and • „The interview process is more likely to select a nice orthodontist than a clever one‟. Less than 10% of the interviewers agreed that:
• „Appointment by application form, without interview, would have selected the same candidates‟,
• „One interviewer is needed at each station‟,
• „The interview process should involve more stations‟, • „The interview was unfair to male candidates‟ and • „The workload involved in interviewing was excessive‟.
Free text comments were given by 15 (21%) of the interviewers. Four interviewers commented about the space for comments on the assessment form and two interviewers were uncertain whether the process selected the best candidate. Two interviewers made suggestions about the logistics of attending the interviews.
The interviewers strongly agreed (mean score of >4.0) with these statements:
„Interview stations were of the right duration‟,
„Interviewers usually agreed on their assessment of the candidate at each station‟,
„My performance over the two days was consistent‟,
„Our interviews appeared fair to the candidates‟,
58 | P a g e
„The same panel conducting the interviews for a second time, several weeks later and with the same candidates, would appoint the same people‟, and
„We tested an appropriate range of competencies during the interview‟. (Table 6) Interviewers agreed that „Performance during the interview helped to predict future performance‟ (mean score 3.9; 95%CI 3.49, 3.22). Ratings suggested that there was a weak agreement (mean score 3.0) with the statements:
„The interview process is more likely to select a nice orthodontist than a clever one‟;
„The interview process should involve more stations‟; and
„The workload involved in interviewing was excessive‟. Interviewers did not agree that
„Appointment by application form without interview would have selected the same candidates‟;
„Only one interviewer is needed at each station‟ and
„The interview was unfair to male candidates‟.(Table 6) There was no clear opinion when considering the statements:
„The candidates‟ personality influenced my assessments‟ or
59 | P a g e
Table 6 Interviewers‟ agreement with and Likert score for statements about the multi-station interview.
No Questions
Agree Disagree Likert scale
N Number Percentage (%) Number Percentage (%) Minimum Score Maximum Score Mean Standard Deviation (95% CI)
1 The interview format selects the best candidate to be
appointed. 32 30 93.8 2 6.2 1 6
4.78
0.97 (4.43, 5.13)
2 We tested an appropriate range of competencies during the
interview. 36 34 94.4 2 5.6 1 6
5.11
1.06 (4.75, 5.47)
3 We asked searching questions during the interview. 36 33 91.6 3 8.4 2 6 4.72 1
(4.38, 5.06)
4 The format was better than that of a traditional panel. 35 29 82.8 6 17.2 1 6 4.6 1.19
(4.19, 5.01)
5 Performance at interview predicts future performance. 34 27 79.4 7 20.6 1 5 3.85 1.05
(3.49, 3.22) 6 The interview process is more likely to select a nice
orthodontist than a clever one. 35 10 25.6 25 74.4 1 5
2.8
1.16 (2.40, 3.20)
7 Our interviews appeared fair to the candidates. 36 36 100 0 0 4 6 5.47 0.7
(5.24, 5.71)
8 Interview stations were of the right duration. 36 34 94.4 2 5.6 2 6 5.36 0.93
(5.05, 5.68) 9
The same panel conducting the interviews for a second time, several weeks later and with the same candidates, would appoint the same people.
36 35 97.2 1 2.8 1 6 5.06
(4.71, 5.40) 1.01
10 Interviewers usually agreed on their assessment of the
candidate at each station. 34 34 100 0 0 4 6
5.35
0.77 (5.08, 5.62)
60 | P a g e
No Questions
Agree Disagree Likert scale
N Number Percentage (%) Number Percentage (%) Minimum Score Maximum Score Mean Standard Deviation (95% CI)
11 Interviewers usually agree when comparing their assessment
of candidates at different stations. 21 21 100 0 0 4 6
4.71
0.78 (4.36,
5.07)
12 I would consistently offer the same assessment if asked to
review my opinion at a later date. 35 32 91.3 3 8.7 1 6
4.8
1.11 (4.42,
5.18)
13 My performance during over the two days was consistent. 35 35 100 0 0 4 6
5.49
0.7 (5.24,
5.73)
14 The workload involved in interviewing was excessive. 35 3 8.5 32 91.5 1 4
2.46
0.85 (2.16,
2.75)
15 Appointment by application form without interview would
have selected the same candidates. 35 3 8.6 32 91.4 1 5
1.83
1.22 (1.41,
2.25)
16 The interview process should involve more stations. 35 3 8.6 32 91.4 1 6
2.11
1.16 (1.72,
2.51)
17 One interviewer is needed at each station. 35 3 8.5 32 91.5 1 6
1.71
1.38 (1.24,
2.19)
18 The candidates’ personality influenced my assessments. 36 22 61.1 14 38.9 1 6
3.39
1.42 (2.91,
3.87)
19 The interview was unfair to male candidates. 34 1 2.9 33 97.1 1 4
1.26
0.71 (1.02,
1.51)
20 The interview helped candidates to understand what a good
orthodontist is. 36 16 44.5 20 55.5 1 5
3.08
1.23 (2.67,
61 | P a g e
Interviewees
Demographic data
All 82 (100%) interviewees completed the questionnaire. Fifty-one (61%) were female and the interviewees‟ age ranged from 25 to 44 years with a mean age of 28.9 years (95%CI 28.1, 29.6). The year of qualification ranged from 1992 to 2009 with over half (68%) having qualified within the last 5 years. Of the interviewees, 45% had not had past experience of a multi-station interview.
Responses to the statements
Agreement
More than 90% of the interviewees strongly agreed with these following statements (Table 5):
„The questions were easy to understand‟,
„I was given adequate information on arrival‟,
„I thought the overall format was fair‟,
„The interview process was well organised‟ and
„I was well informed about the interview process‟.
More than 75% but less than 90% of the interviewees agreed that:
„I thought that the multi-station interview was fairer than a traditional panel interview‟ (Figure 4),
„I felt well prepared for the types of questions asked‟ and
„I found the multi-station interview helpful to me‟.
73% of the interviewees preferred the MSI to a traditional panel interview (Figure 6). The mean Likert scale score for how „fair‟ the interviewees found this interview format was 4.76, (SD 0.98, 95%CI 4.54, 4.94) (Table 7).
62 | P a g e
Table 7 Interviewees‟ agreement with and Likert score for statements about the multi-station interview.
No Question
Agree Disagree Likert Scale
N Number Percentage (%) Number Percentage (%) Minimum Score Maximum score Mean (95% CI) Standard Deviation
1 I was well informed about the interview
process. 82 74 90.3 8 9.7 2 6
4.8
1.07 (4.57,
5.04)
2 I found the multi station interview helpful to me. 81 66 81.7 15 18.3 2 6
4.46
1.1 (4.21,
4.70)
3 I was given adequate information on arrival. 82 77 93.9 5 6.1 2 6
4.98
0.94 (4.77,
5.18)
4 The questions were easy to understand. 81 79 96.3 3 3.7 2 6
4.54
0.78 (4.37,
4.71)
5 I felt well prepared for the types of questions
asked. 82 67 81.9 15 18.1 2 6
4.28
0.88 (4.09,
4.47)
6 I thought the overall format was fair. 82 77 93.9 5 6.1 1 6
4.76
0.98 (4.54,
4.97)
7 I prefer this format to a traditional panel
interview. 81 67 73.2 14 26.8 1 6
4.22
1.36 (3.92,
4.52)
8 I thought that the multi-station interview was
fairer than a traditional panel interview. 82 71 85.5 11 14.5 1 6
4.63
1.13 (4.39,
4.88)
9 The interview process was well organised. 82 76 91.6 6 8.4 1 6
4.94
1.1 (4.70,
63 | P a g e
Figure 3 Interviewers' perception of „Our interviews appeared fair to the candidates‟.
64 | P a g e
Figure 5 Interviewers' perception of „The format was better than that of a traditional panel'.
65 | P a g e
Free text comments were given by 42 (51.2%) interviewees. Eighteen of the comments (22%) were about the short notice the interviewees received for attending the interviews. Three candidates wanted more time for the questions and two interviewees commented that the delay between when the interview took place and when the results of the interview were released to them was too long.(Table 8)
Table 8 Interviewees‟ comments from the Phase I survey
1. Old system fairer; 2 months wait for results is not acceptable 2. Re Q12, all ok except research Q.
3. Re Q12, not for research Q, difficult to tell what exactly they were after 4. Short notice btw notification of interview n interview date
5. Short notice prior interview, esp regarding references.
6. Short notice; Doc to bring unspecified, eg.photos n p/copy of passport. 7. Short notice; fairer sys; length of waiting too long, unfair for other jobs.
8. Some Qs were very vague
9. Time too short for Q; Too long delay for result, how to tell hosp?? 10. Too short notice
11. Too short notice for the interview and getting the p/folio together. 12. Too short notice for the interview and getting the p/folio together. 13. Too short notice prior interview
14. Too short notice(5 days b4 interview), pt cancellation 15. Too short notice, 3 workings days!
16. Too short notice, a wk not enough to gather references and p/folio. 17. Too short notice, in rush to fill up forms n Ref need weeks to sought. 18. Too short notice; all eligible had interview, ?CV impact; Consultants say… 19. Understandable; written station too long; scenario in separate sheet. 20. Well structured day; Everyone is very helpful.
66 | P a g e