La semiótica de la cultura y el concepto de texto

The rationale for this study was to examine the efficacy of the relationship between selection variables and assessment variables in surgical training. A goal is to optimise selection methods so those with the greatest likelihood of succeeding in surgical training and in the practice of surgery are most likely to be selected into the Royal Australasian College of

Surgeons (RACS) surgical education and training (SET) program. The investigator performed a structured review of records and multiple-year analysis of performance in selection and

assessment measures for this quantitative study.

Research questions

Main question

What is the relationship between scores at selection and scores in assessments during the first two years of surgical training?

Subordinate questions

The main research question is supported by the sub-questions:

What are the performance characteristics of each of the selection items? What are the performance characteristics of each of the assessment items? To what extent do selection scores predict early assessment scores?

Study design

The study is a correlational design, assessing relationships between selection scores and subsequent scores for in training assessment. The study used quantitative data to analyse relationships between scores in selection and scores in assessments during training for a sample of trainees undertaking surgical education and training in General Surgery (GS) at the Royal Australasian College of Surgeons (RACS). Quantitative data drew on surgical trainees’ scores

in specified selection and assessment items. Scores at each sample point were compared to determine: 1. The strength of interrelationships between scores in each of the selection items; 2. The extent to which selection scores predict assessment scores; 3. The strength of correlations between scores in selection and assessment items; 4. Similarities and differences between three annual cohorts of trainees; 5. Similarities and differences between Australian and New Zealand trainee cohort selection and assessment performance.

A retrospective, longitudinal design was used, tracking GS trainees in three cohorts— selected in 2008-2010—across the first two years of training. Pearson product-moment

correlations and regression analyses were used to determine interrelationships between selection instruments and the predictive validity of the selection instruments. Examples of similar design methodology are described in many studies, including those undertaken by Carmichael et al (2005), Dirschl et al (2002), Eva et al (2009), Patterson, Lievens, Kerrin, Munro and Irish (2013), Poole, Shulruf, Rudland and Wilkinson, (2012), Reiter, Eva, Rosenfeld and Norman (2007) and Sladek, Bond, Frost and Prior (2016).

The study population

Trainees enrolled in RACS SET program in GS constituted the study population. Those within the study population are referred to variously as ‘applicants’, ‘candidates’, ‘surgical trainees’ or ‘trainees’. The terms ‘participants’ and ‘subjects’ are not used as no recruitment was undertaken to establish the study sample. The group of trainees who were selected in any specific year are referred to as that year’s ‘cohort’.

The study reviewed selection and assessment archival data for three entire cohorts of Australian (Au) and New Zealand (NZ) GS trainees. Data was reviewed for 347 Au and NZ GS trainees. The three consecutive cohorts that formed the basis of the study applied to RACS in 2008 (n = 100), 2009 (n = 107) and 2010 (n = 140), commencing surgical training in 2009, 2010 and 2011 respectively. The trainees were located throughout Au and NZ—in all major cities and in rural and remote locations. At the time of the study most of those selected in 2008 had

progressed into their third year of training (SET3), with the 2009 and 2010 cohorts in SET2 and SET1 respectively.

The instruments

The instruments used to ascertain candidate performance in GS selection to surgical training comprised three selection instruments and total selection scores. The surgical trainee performance assessment items comprised three examination items and three work-based assessment items. These are identified in Figure 6. The assessment scores used in the study comprised each trainee’s results in: three selection activities—structured Curriculum Vitae (CV), structured referee reports (RR), structured multi-station interviews (Int)—and a total selection score (Total sel); their first attempt at three examinations—the Generic surgical science examination (GenSSE), the Specialty surgical science examination (SpecSSE) and the Clinical examination (CE)—and ratings in four implementations of each of the three in-training and work-based assessment items—the Direct observation of procedural skills (DOPS), the Mini Clinical Evaluation Exercise (MiniCEX) and the End of term assessment (ETA). See Figure 6. Trainee performance scores in all instruments were converted to percentages.

Selection score data was collected in all selection items and assessment performance data was collected for all available assessments. Candidates’ scores were calculated for each selection item, were converted to percentages, and were recorded; an aggregate Total selection (Total sel) percentage score was derived for each candidate. Assessment scores were recorded for trainees’ first attempts in each of the three examinations (GenSSE, SpecSSE, CE). Work based assessment reports (DOPS, MiniCEX and ETA) for all trainees for each rotation were reviewed and ratings were converted to numeric scores for each report; scores for sub-

components within the ETA reports were also calculated. See Table 1 for DOPS, MiniCEX and ETA scoring conversion parameters. These instruments were chosen to provide the data for this study, as they comprise the key, quantifiable indicators that are used in the GS training program to measure trainee performance.

Table 1 Conversion of assessment ratings to numeric scores

Assessment Maximum score Rating category Numeric score

DOPS (form A) 30

Unsatisfactory 0

Borderline 1

Competent 2

Excellent 3

Not observed/Not applicable Maximum score

reduced by 3 points DOPS (form B) 30 Unsatisfactory 0 Borderline 1 Competent 2 Excellent 3 Significant improvement required 1

Some improvement required 2

Competent 3

Not observed/Not applicable Maximum score

reduced by 3 points MiniCEX 27 Unsatisfactory 0 Borderline 1 Competent 2 Excellent 3

Not observed/Not applicable Maximum score

reduced by 3 points ETA 142 Competencies Not competent 1 Borderline 2 Competent 3 Excellent 4

Not observed/Not applicable Maximum score

reduced by 4 points Essential criteria

Unsatisfactory 1

Satisfactory 2

Not observed/Not applicable Maximum score

Independent variables.

The independent variables used in this study comprised scores in the three selection instruments—CV, RR and Int—that were used by RACS to rank candidates for offers to enter surgical training. Summed scores in these three selection instruments formed candidates’ total selection scores. These selection instruments suited the research model and provided the entirety of available information about candidates’ performance in the selection processes. Each selection instrument utilised different methods to identify and score candidates’ attributes. The study could therefore assess the relationship between each independent variable and the independent variables. No candidates had missing scores in selection instruments.

Dependent variables.

In accord with literature arising from previous studies, assessments undertaken during training were used as outcome variables. Assessments used in this study were limited to those undertaken in the first two years of training, comprising DOPS, MiniCEX, ETAs and

examinations. These assessments are summarised in Figure 6. The RACS GS assessments comprised both formative, work-based assessments and summative examinations. Information from all but one assessment implemented in the first two years of surgical training was used. The assessment not used in the study was a work-based, formative, mid-term assessment (MTA), that was administered part way through surgical rotations. As the content of the MTA and the ETA was identical, and trainee performance in the two assessments reflected this, it was considered that no benefit would attract to reviewing both assessments. It was therefore decided to only include the ETA as it filled a more critical role and had higher completion rates,

possibly due to its dual formative and summative aspects.

Comparable studies

Other studies have used similar methodologies to analyse relationships between scores in selection and scores in assessments in primary medical and specialty training. Many

Selection instruments

• Structured Curriculum Vitae (CV) • Structured referee reports (RR)

• Structured multi-station interview (Int) • Total selection score (Total sel)

Assessment instruments

• Examinations

o Written Examinations

• Generic Surgical Sciences Examination score (GenSSE)

• Specialty specific Surgical Sciences Examination score (SpecSSE)

o Practical Examination

• Clinical Examination score (CE)

• Work-based assessments

o Direct Observation of Procedural Skills (DOPS)

• Direct Observation of Procedural Skills 1 (DOPS1)

• Direct Observation of Procedural Skills 2 (DOPS2)

• Direct Observation of Procedural Skills 3 (DOPS3)

• Direct Observation of Procedural Skills 4 (DOPS4)

• Direct Observation of Procedural Skills Average score (Average DOPS)

o Mini Clinical Evaluation Exercise (MiniCEX)

• Mini Clinical Evaluation Exercise 1 (MiniCEX1)

• Mini Clinical Evaluation Exercise 2 (MiniCEX2)

• Mini Clinical Evaluation Exercise 3 (MiniCEX3)

• Mini Clinical Evaluation Exercise 4 (MiniCEX4)

• Mini Clinical Evaluation Exercise Average score (Average MiniCEX)

o End of Term Assessment (ETA)

• End of Term Assessment Report 1 (ETA1)

• End of Term Assessment Report 2 (ETA2)

• End of Term Assessment Report 3 (ETA3)

• End of Term Assessment Report 4 (ETA4)

• Average End of Term Assessment Report (Average ETA)

Project approval and ethics approval

The study was conducted with the approval of RACS. In 2009 the RACS Board of Surgical Education and Training (BSET), which monitored and coordinated activities associated with the nine surgical training programs, approved access to trainee records for this study. In the same year, approval was gained for the study from the Victoria University Ethics

Committee and the RACS Ethics Committee, which recommended appointment of a RACS representative as a co-supervisor, ensuring that RACS was apprised of project developments and outcomes.

It was not deemed necessary to gain consent from the trainees to review and analyse data pertaining to selection and assessment scores as: all data were archival, held by the Board in General Surgery (BiGS) and by RACS, under whose auspices the study was conducted. I was an employee of RACS at the time of the study; a condition for approval of the study was to de- identify all data when reporting results.

Ethical issues encountered during this study primarily reflected the confidential nature of the data, which were comprised of candidates’ scores in selection items, work-based

assessments and examinations. Although individuals’ names were recorded when data were collected, care was taken to de-identify all data during analysis and reporting. An additional ethical concern has arisen in reporting on the content of the selection and assessment

instruments. It is not possible to reproduce detailed interview or examination content in this study as this content is not in the public domain and these selection and assessment items may be re-used subsequent to this report. Articulating the content of these items may adversely affect their future validity, reliability and fairness.

Psychometric characteristics of the instruments

The validity, reliability and predictive capacity of the selection instruments—also called selection tools—used by RACS GS had not been established prior to the current study. This

study provides information to contribute to RACS’ understanding of the current and potential uses of these selection instruments.

Selection instruments.

Selection processes and instruments were specified in annual GS Au and GS NZ selection regulations, which were subject to approval by the RACS BSET. See Appendix C for Selection to surgical education and training in General Surgery regulations, 2008, 2009, and 2010.

Decisions about appointment into RACS GS SET in both Au and NZ were based entirely on candidates’ performance in three selection activities: the structured Curriculum Vitae (CV), structured referee reports (RR) and structured multi-station interviews (Int). Each of the selection instruments constituted a defined proportion of the total selection score (Total sel). Although the CV, RR and Int were used in each annual selection process, iterative differences in the content of these instruments and modifications to protocols occurred annually. Differences between the content and implementation of Au and NZ instruments also transpired.

The three selection instruments cumulatively collected information about candidates’ experiences and attributes. The CV recorded applicants’ self-reported, authenticated biographic information, clinical experience and academic and personal accomplishments (Oldfield,

Beasley, Smith, Anthony & Watt, 2013); these were categorised as medical expertise and technical expertise; scholar and teacher; and management and leadership. The RR scored applicants’ workplace performance, as judged by their supervisors, in criteria aligned to the RACS competencies of collaboration; communication; judgement and clinical decision making; management and leadership; medical expertise; professionalism; scholar and teacher; and technical expertise. In 2008, GS Au used a fourth selection instrument, the Hospital

Assessment Report (HAR). The HAR content was identical to the RR, but required hospital personnel other than surgeons as assessors. The Int assessed candidates’ contribution to GS and personal attributes as they related to RACS competencies: communication and collaboration; health advocacy and cultural awareness; management and leadership; professionalism; and scholar and teacher. See Appendix D for examples of the CV and RR.

Applicants were initially scored against criteria in the CV and RR; candidates’ percentage scores in these two instruments were combined and those whose scores surpassed a designated minimum score were invited to interview. Candidates’ Int scores were combined with the CV and RR scores to form an aggregate total selection score (Total sel). Candidates were ranked by their Total sel, with those achieving the highest scores being offered positions in the GS SET program. The number selected varied each year, with the intake dependent on the number of training positions—or ‘posts’—available. The number of available training posts per year was a factor of the number of extant trainees who vacated training positions—usually this was by progression through the program or completing the program; less frequently by withdrawing or being dismissed from the program.

GS allocated proportional ‘weightings’ to the three selection instruments within ranges specified by RACS’ BSET and consistent with other surgical specialties conducting training under the auspices of RACS. All specialties implemented the same three selection instruments, within the weighting ranges indicated in Table 2.

Table 2 Selection instruments and weightings

Selection instrument Weighting

Curriculum Vitae 15%–25% Structured referee report 35%–45%

Interview 35%–45%

The information to be gathered by each selection instrument, the allocation of a maximum score per selection item and the protocols for implementing selection instruments were determined independently by each surgical specialty, to suit their requirements. Selection instrument weightings implemented by GS in 2008, 2009 and 2010 are presented in Table 3.

Table 3 GS selection instruments with proportional weightings per year

Structured CV Structured referee

reports

Structured multi- station interview

Total selection score

Specified weighting range 15%–25% 35%–45% 35%–45% 100% Year 2008 2009 2010 2008 2009 2010 2008 2009 2010 2008 2009 2010 General Surgery Au 20% 20% 20% 30%a _{40% 40% 40% 40% 40% 100% 100% 100%} General Surgery NZ 20% 20% 20% 40% 40% 40% 40% 40% 40% 100% 100% 100%

Note. Data sourced from Selection to surgical education and training in General Surgery regulations (2008, 2009, and 2010).

a. In 2008 the GS Au RR was weighted at 30%, with an additional selection tool, the Hospital Assessment Report (HAR), weighted at 10%. The HAR was not reviewed in this study; instead, the weighting of the 2008 GS Au RR is considered to be 40%.

Structured Curriculum Vitae (CV).

The RACS GS Curriculum Vitae (CV) CV gathered biographic information pertaining to candidates’ clinical experience and academic and personal accomplishments. Collins (2007) states that “the purpose of the [RACS’] CV is to enable applicants to provide a synopsis of their qualifications, meritorious performances, appointments and experience in various areas of medical and surgical practice” (p. 11). Collins (2007) recommends that the marks allocated in the CV “should ensure a balance between clinical experience and academic achievements” (p. 12). The precise information gathered in the CV varied between surgical specialties, as did the value placed on each component, the scoring of CV content and the weighting of this instrument within the selection process. Components in the GS CV, in the years under review, included: surgical experience, skills, qualifications, presentations, publications, prizes and leadership. Scoring allocations for these components varied per year. CV forms were included in the application form in 2009 and 2010. CV information for 2008 is included in Appendix D.

Applicants self-reported their biographic information on online application forms. Documented proof of all accomplishments recorded in the CV was required. This objective information, comprising itemised lists of verified activities, was scored according to

predetermined scales, with a fixed maximum score allocated to each category. Two assessors scored the CVs, adhering to a scoring rubric. Each CV score was usually calculated as the mean of the two assessors’ scores; if there was a disparity between assessors’ scores, the GS

chairperson re-scored CV. CV raw scores were converted to percentage scores. Any applicants found to provide unverified or false information were removed from the selection process. Information included by candidates in their applications was only scored if it was pertinent to the defined CV categories. No additional information beyond that specified in the CV was accepted. Annual scoring components and maximum allocated scores in the CV are presented in Table 4.

Structured referee reports (RR).

Structured referee reports (RR) were implemented to assess applicants’ workplace performance—their “personal attributes, quality of work and suitability for the SET Program” (Selection to surgical education and training in General Surgery regulations, 2008, p. 7; 2009, p. 6; 2010, p. 6), as judged by their supervising consultants, in criteria aligned to the RACS competencies of collaboration; communication; judgement and clinical decision making; medical expertise; professionalism; scholar and teacher; and technical expertise. Referees rated applicants against competency statements. RR content and methods of identifying referees were reviewed annually by the (Australian) Board in General Surgery (BiGS) and the New Zealand Association of General Surgeons (NZAGS), with changes subject to approval by RACS’ BSET.

Table 4 CV components and maximum possible scores in 2008, 2009 and 2010

Competencies, experiences and accomplishments Maximum

score Au and NZ 2008a Maximum score Au and NZ 2009b Maximum score Au and NZ 2010c Medical expertise and technical expertise

Surgical and medical experience 8 points 5 points 7 points

Skills courses 2 points 2 points 2 points

Scholar and teacher

Qualifications (higher degrees, beyond initial medical degree) 3 points 6 points 4 points

Presentations 4 points 4 points 5 pointsd

Publications 4 points 4 points

Prizes and awards 2 points 2 points 2 points

Scholar and teacher (teaching) - - 3 points

Management and leadership

Leadership 2 points 2 points 2 points

Total score possible 25 points 25 points 25 points

a Data sourced from Selection to surgical education and training in General Surgery regulations (2008). b Data sourced from Selection to surgical education and training in General Surgery regulations (2009). c Data sourced from Selection to surgical education and training in General Surgery regulations (2010). d In 2008 and 2009, presentations and publications were scored separately; in 2010 presentations and

publications were combined into a single score.

Referees.

During the years under review in the current study, each applicant nominated two

supervising consultants per rotation, with whom they had worked during the previous two years (2008) or four years (2009 and 2010). The specialty contacted three of these consultants per applicant in 2008 (Selection to surgical education and training in General Surgery regulations, 2008), and five consultants per applicant in 2009 and 2010 (Selection to surgical education and training in General Surgery regulations, 2009; 2010), sending each referee a structured referee report form to complete. Referees submitted their reports to BiGS, or to NZAGS, who allocated

scores, based on referees’ ratings. In 2008, applicants required three valid referee reports in order to proceed in the selection process; five valid reports were required in 2009 and 2010. Referees were not trained in assessing candidates, nor in completion of the reports. A summary of requirements regarding numbers of referee nominations and referee reports per year is presented in Table 5.

Table 5 Number of referees and referee reports used in 2008, 2009 and 2010 Minimum requirements

per applicant

2008a 2009b 2010c

Au NZ Au NZ Au NZ

Number of nominated refereesd ₈e _{8 10}f _{10 10}g _8-10g

Number of referee reportsh 3 5 5 5 5 5

a Data sourced from Selection to surgical education and training in General Surgery regulations (2008). b Data sourced from Selection to surgical education and training in General Surgery regulations (2009).

In document La enseñanza "audiovisual" del inglés como lengua extranjera (página 62-67)