LOS ECOSISTEMAS ● Ecosistema: identificación

“No single assessment method can provide all the data required for judgment of anything so complex as the delivery of professional services by a successful physician” (Miller 1990). Indeed the objective measures of skill performance in endovascular intervention are poorly reported. A number of assessment tools are available. These include time-action analysis, motion analysis, VR simulator automated parameters, task specific checklists (TSC) and global rating scores (GRS).

1.10.1 Time-Action Analysis:

Time-action analysis involves breaking procedures down into segments and analysisng the time it takes the trainee to perform each step of a procedure. Although the technique has been adopted in minimally invasive training studies

(Bakker et al 2002) it is both time consuming and considered a poor measure of overall procdure quality.

1.10.2 Motion Analysis

Analysing economy of movement and purposeful motion is a more discriminatory way of assessing technique and the overall quality in technical skill performance. (Bann et al 2003) This particular method is in fact used to assess candidate’s dexterity during open vascular skills assessment in the EBSQ-VASC using a motion-tracking device. The Imperial College Surgical Assessment Device is capable of measuring candidate’s economy of movement by tracking hand- movement with a electromagnetic sensor. The model has been used to assess laparoscopic surgery (Smith et al 2002), but not, to date, used in assessment of endovascular skills performance.

1.10.3 VR Measurable Parametres

VR parameters have previously been described (see 1.5.3). They are automated scores derived from the VR models instantaneously following a candidates performance. A number of studies, looking a various endovascular interventions have shown construct validity based on VR measured parameters alone. Dayal et al looked at twenty-one trainees of varying levels of endovascular experience, performing CAS using the Procedicus VIST simulator. Analysis of their VR parameters demonstrated the models construct validity. (Dayal et al 2004). Following criticism aimed at studies including medical students (who lack the baseline knowledge of basic endovascular skills), Van Herzeele et al looked at qualified practitioners only and discovered a similar pattern. Experienced practitioners are quicker and use less fluoroscopy when completing a CAS procedure on the VIST simulator, compared to less experienced practitioners (Van Herzeele et al 2007).

It is widely accepted that increased speed does not necessarily translate to a performance that confers greater safety for patients (Patel et al 2006). Hislop et al highlighted this exact point explaining that innate endovascular aptitude, represented by time to complete a performance, can be improved with non- endovascular training such as video games (Hislop et al 2006).

1.10.5 Task Specific Check Lists and Global Rating Scales

Traditionally trainers made global rating assessments of trainees performance in the subjective assessment of their competency to perform a certain procedure. Such judgments are often unreliable measures of true performance (Streiner 1985). Research to identify a more reliable and standardized method of assessing technical skill led to the development of procedure specific checklists and global rating scales. Kopta developed one of the first checklists to assess orthopaedic trainees operative skills in 1971 (Kopta (b) 1971), and Schueneman et al used a rating scale which was able to differentiate trainees of various experience (Schueneman et al 1984).

Despite first being reported by Martin et al, the Objective Structured Assessment of Technical Skill (OSATS) model has become synonymous with the Canadian physician Richard Reznick (Martin et al 1997). It was developed to address the lack of standardisation seen in operations used for assessment, variations in examiners standards, and trainee performance. OSATS consists of two components: a task specific checklist (TSC), which breaks procedures down into a series of steps, and a global rating scale (GRS). The GRS is a quantitative assessment measure of technical skill based on seven aspects of performance, each scored on a Likert scale from 1 to 5. It includes parameters such as “respect for tissue”, “flow of operation”, and “instrument handling” (Martin et al 1996). The two components are necessary, as the TSC identifies where an error took place, and the GRS provides an objective score of overall performance.

OSATS was developed to assess the open surgical procedure performance, and has been shown to have construct validity in the assessment of open (Nielsen et al 2003) and laparoscopic (Eubanks et al 1999) technical procedures. It has also been adopted as the gold standard assessment tool for technical skills assessment in endovascular literature (Martin et al 1997, Chaer et al 2006, Hislop et al 2006, Berry et al 2007, Tedesco et al 2008, Van Herzeele et al 2009, Berger et al 2010, Riga et al 2010,). Hislop et al used a modified OSATS tool, calling it a modified Reznick scale (MRS) and showing it to have construct validity in their trial of practitioners performing carotid intervention (Hislop et al 2006). It was modified and formed the assessment tool of choice in Berry et al’s porcine transfer study (Berry et al 2007). Chaer et al included nine items in their modified GRS, including parameters to measure “wire and catheter handling”, “awareness of wire position”, “maintenance of wire stability”, “awareness of fluoroscopy usage” and “precision of wire/catheter technique” Chaer et al 2006).

Interestingly to date TSC and GRS have not been used for the assessment of endovascular performance in the interventional operating suite in real patients. Indeed Beard and colleagues (Beard J 2011) performed a prospective observational study of the methodology for assessment of surgical skills with the aim of comparing user satisfaction, acceptability as well as the reliability and validity of OSATS and Procedure Based Assessments (PBAs). They concluded that in fact PBAs have a higher utility for assessing technical skills observed in the operating theatre compared to OSATS. It is therefore acknowledged that although OSATS has been used in this thesis for technical skill assessment, the PBA may be a more appropriate method of assessment when operating on real patients.

1.10.6 Utility Index

The utility index is a concept of clinical performance assessment consisting of six components (Van Der Vleuten 2006): Educational x validity x reliability x cost x acceptability x feasibility. There is a growing appreciation that no single assessment tool (OSATS or PBAs) can adequately assess the clinical performance of a trainee. Indeed ‘assessment planning should focus on assessment systems

with triangulation of data in order to build up a complete picture of a doctor’s performance’ (PMETB 2007)