Cuestionario a expertos

In order to pilot the Improvement Type Measure, some descriptions of improvement projects were required, against which ratings could be made. Examples of improvement work undertaken by THF Award Holders were extracted from the prior documentary review and were formed into brief descriptive paragraphs, as shown in Appendix 6.

Some internal piloting of the ITM was undertaken within the research team, using the written scenarios as a basis. The first external pilot of the typology took place at the

end of a workshop for THF Leadership Fellows in York in June 2008. 17 people took part, including one member of THF staff and 2 Leadership Development Consultants. The group was asked to read through each improvement description, as shown in Appendix 6, and to use the Improvement Type Measure instrument (Appendix 5) to rate each one. Participants were also asked, in a focus group, to verbally comment on the pilot instrument in terms of its face validity and its utility.

In his critical examination of the concept of ‘face validity’, Mosier (1947) identifies ambiguities surrounding the use of the term. From the four interpretations he identifies, reference here to face validity in respect of the Improvement Type Measure refers to what he calls Appearance of Validity, namely,

‘a test which is to be used in a practical situation should, in addition to having pragmatic or statistical validity, appear practical, pertinent and related to the purpose of the test as well’ (p.192)

He continues by clarifying that,

‘This usage of the term assumes that face validity is not validity in any usual sense of the word but merely an additional attribute of the test which is highly desirable in certain situations.’ (p.192)

In relation to the piloting of the Improvement Type Measure, the focus group of participants taking part in the pilot were asked about the extent to which the dimensions on the pilot instrument appeared to measure the sorts of aspects which ‘from ordinary experience’ (Roth, 1995, p.390) might be expected in a tool with this purpose.

The data collected from this pilot were manually recorded and analysed. The main findings and feedback are shown in Figure 3 :

 Not sufficient information in the scenario descriptions to accurately rate each item – most common comment

 Very little consistency in ratings obtained from 15 completed questionnaires (2 questionnaires incomplete)

 Some of the polarities are too complicated, conflating more than one element (e.g. items 5 & 7)

 Scenarios 2, 5 & 7 are not improvements – they are studies, therefore the ITM is difficult to apply

 Regarding item 6, even if an improvement is an imposed imperative, the implementation can still be creative

 Limited knowledge of clinical areas amongst some respondents limited their ability and confidence to rate the improvements.

Figure 3. Verbal feedback from respondents piloting the ITM, June 2008

The lack of sufficient information to make a rating appeared to be an underlying factor contributing to the inconsistency of ratings in the pilot at York, and was therefore a prime area of focus in refining the methodology. The author refined the descriptions of improvement work, replacing the 7 brief summary paragraphs with 3 more detailed descriptions, as shown in Appendix 7.

In addition, some changes were made to the items on the Improvement Type Measure, in response to feedback from the first pilot. The wording at each end of the visual analogue scale was simplified, and the dimension measuring the scale and complexity of influencing stakeholders was divided into two separate items. In response to specific feedback from pilot participants, the dimension measuring patient

impact was divided into two separate items, to allow for health outcome and patient experience to be rated separately. This created an Improvement Type Measure with 9 dimensions (Appendix 8). The pilot of this second version was run at a lunchtime workshop with THF staff. Twenty-two completed questionnaires were returned.

Despite the more detailed examples, specifically written to contain information relating to each item on the Improvement Type Measure, analysis of data from the second pilot showed only slightly better reliability than with the first version. The most concrete scenario (cleft lip and palate network) had the most reliable consistency, and the most ephemeral scenario (high impact changes) the least consistent, but in both cases, the spread of ratings showed that in its current form, the ITM was far from being a reliable measure.

Respondents in the two pilots had provided positive verbal feedback during the focus groups about the face validity of the dimensions. However, there was a sense that whilst the face validity was good, the utility of the instrument as a self-assessment tool was potentially becoming reduced by its intricacies.

The researchers spent many hours debating the tension between developing a holistic, qualitative tool on the one hand and a highly precise, quantitative but reductionist instrument on the other. The aim was to design a typological instrument whereby reliability, validity and utility could be optimised.

Given the difficulties encountered by pilot respondents from a lack of sufficient information in a written scenario, a paper-based approach to classifying improvement work was looking increasingly impractical. It became apparent at this stage that the

essence and detail of improvement work required for useful classification could only be captured through conversation and verbal explanation. Consequently, it was decided that the Improvement Type Measure would be developed into a semi- structured interview format, which could be used with individual THF Award Holders. This would then be incorporated into the semi-structured interview schedule as part of the data- gathering stage of the research.

The work to develop the ITM upto this point was presented and shared at a seminar run by the THF, involving academic advisers, the researchers, senior THF managers and Leadership Development Consultants in July 2008. The seminar provided an opportunity to reflect on the work of the research team thus far, to scrutinise the approach taken and decisions made, and to offer peer review on the overall process and progress. As a result of discussions during this seminar, the decision was taken to change the Improvement Type Measure from a self-assessment instrument into one that had to be administered by trained ‘experts’ (ie the researchers and people trained by them). Clearly, such a move made the wider dissemination of the instrument harder to envisage, but it did allow the notion of a more sophisticated, detailed instrument to be developed, whose utilisation could potentially have benefits beyond the simple assessment of the complexity of an improvement initiative, and into the realms of a mature developmental tool.

Once this decision had been made, the focus was to improve the reliability of the interview-based ITM instrument. Pilots of this instrument took the form of recorded and transcribed interviews with two previous THF Leaders for Change Award Holders.

The transcripts were then independently rated by each of the two researchers undertaking these interviews against the ITM, and the two of them then met together and with the rest of the research team to discuss the results. On the basis of these discussions, it became apparent that a core method of interpreting each piece of improvement work was developing among the evaluation team, leading to a good level of internal consistency in rating types of improvement work.