Configuración de los equipos

The study followed a sequential design, ordered to meet objectives 1–3 in turn.

Stage 1 covered the development of the new measure. This used an adapted form of the ProMES

methodology (adapted to allow development of a more generic measure across a large number of teams), with a series of workshops with general practice staff and public representatives (stage 1a), followed by a consensus exercise to finalise some details of this measure and create an online tool for its use (stage 1b).

Stage 1a: ProMES-based workshops

The ProMES methodology was introduced inChapter 2,Methods. In its traditional form, it involves a series of facilitated workshops with members of a team, in three broad stages: (1) definition/clarification of team objectives, (2) identification of indicators of these objectives and (3) the formation of contingencies to convert these indicators into effectiveness scores. In this study, the traditional ProMES methodology was adapted to be used across an entire sector (general practice) rather than a single team. This followed the approach used successfully in a different NIHR project,82_{which used an equivalent design to develop a} measure of effectiveness for CMHTs.

The design involved three phases of workshops.

Phase 1

Two large, full-day workshops (one in Sheffield and one in London), each aiming to include

30–40 participants (15–20 general practice staff and 15–20 members of the public). These workshops featured a series of group exercises–some kept staff and public separate, and others mixed them up– designed to tease out what theobjectivesof a general practice are. The findings from these workshops were collated and coded, and thematic analysis was used to derive a set of objectives that broadly represented the different views expressed in the workshops.

Phase 2

Six smaller, half-day workshops, spread across multiple locations, each involving six to eight participants. Each workshop’s participants would be more homogeneous in nature (two workshops of GPs, two of practice managers, one of practice nurses and one of public representatives), and would consider potential

indicatorsto measure up to four of the objectives identified in phase 1. Overall, each objective would be considered in at least two different workshops. Participants were strongly recommended to consider using existing data as far as possible. The collation of findings from these workshops would then result in a set of potential indicators, that would be put forward for the phase 3 workshops.

Phase 3

Two large, full-day workshops (again, one in Sheffield and one in London). These were similar in size to phase 1 (30–40 participants, split between general practice staff and members of the public). These workshops would undertake a range of activities with three main aims: (1) to review the indicators

proposed in phase 2, (2) to weight the importance of each of the objectives in order to allow creation of an overall index, and (3) to derivecontingenciesfor each indicator, the ProMES term for functions that convert values of the raw indicator into points contributing to an overall effectiveness score. This effectiveness score would then form the numerator (quality-adjusted outputs) of the overall productivity index, with the denominator (inputs) being represented by the extent of financial resource available to a practice over the period in question.

Precise methods used for each of these phases of workshops are given inChapter 4.

Stage 1b: finalisation of the measure

There were two main elements to this phase: a consensus exercise, and the creation of an online version of the tool that could be used for piloting:

1. Consensus exercise. The aim of the consensus exercise was to build agreement on the weightings of the different indicators/objectives chosen, and to determine whether or not the measure appeared feasible (in terms of the data collection required) and valid (i.e. that it appears to measure something that is accepted as a measure of productivity). This included an initial meeting, to which representatives from relevant NHS, regulatory, professional and patient bodies were invited. This was followed by a series of online communications, including both e-mails and an online questionnaire, to finalise details. 2. Creation of an online tool. The tool used an online platform named Effecteev®_{designed for ProMES}

applications by the specialist German company Feedbit Software GmbH (Nürnberg, Germany), to assist teams with the ProMES process. This software was used during stage 1a also, to help develop the contingencies. As part of this process, guidance (both online and in document form) was developed to instruct/assist practices in using the tool.

In reality, the precise methods for stage 1 deviated somewhat from the above for a combination of practical reasons. These variations, and their implications, are detailed inChapter 4.

Stage 2 involved the piloting of this measure. The aim was to recruit 50 general practices from a variety of CCGs and regions to use the measurement tool over a 6-month period (stage 2a). This would then be evaluated (stage 2b) to test whether or not the tool worked in a way that was feasible for practices, whether or not it was perceived to be useful and whether or not there was any evidence of improvements being made with its use.

Stage 2a: piloting of the measure

A target was set for 50 practices to be recruited to trial the measure and online tool over a 6-month period. The aim was that the 50 practices should be spread across 8–10 CCGs, some of which would also have participated in stage 1 of the study. CCGs were recruited purposively, so that those involved in stage 1 were invited first, and then others selected to ensure a balance across regions of England and a balance between those that were more urban or more rural in nature. CCGs that agreed to participate sent out a

OVERALL METHODS

NIHR Journals Library www.journalslibrary.nihr.ac.uk 28

request to their general practices for expressions of interest in the study. Practices that had participated in stage 1 were also contacted directly by the researchers.

A total of 51 practices agreed to participate. These were targeted to ensure that these practices had a good spread in terms of location, size and characteristics of the local population. Each of these 51 practices was visited by a member of the research team or a trained Clinical Research Network (CRN) representative, and at least one member of staff in the practice was trained in using the online tool, including details of the data collection required, and retrieving and interpreting monthly reports.

Each practice would then use the tool for a period of 6 months: collating and entering data on a monthly basis, and generating a report to indicate how indicators, objectives and overall effectiveness were changing over time. This would be followed by the evaluation. Following completion, practices were each paid £500 in compensation for the time spent piloting the tool.

Stage 2b: evaluation of the measure

At the end of the 6-month period, the measure was evaluated. The evaluation took three forms:

1. Analysis of effectiveness data. A statistical analysis of monthly practice data was conducted to determine whether or not there was any evidence of change over time overall, change over time for specific objectives or change over time for specific indicators, and whether the extent of change was related to characteristics of the practice.

2. Interviews with practice representatives. Telephone interviews were conducted with representatives from each practice. These were, in some cases, the same practice managers who had responded to the questionnaire, but, in other cases, GPs or other practice staff were interviewed (this was spread out to give a more balanced view). Interviews covered the perceived usefulness of the measure (including how widely results were discussed in practice meetings), whether or not there were any areas not covered, and what might be done to improve the usability of the tool. In some cases, interviews (and two focus groups) were conducted with patient representatives; these only covered the usefulness and coverage of the measure, particularly focusing on those sections most relevant to patient experience.

3. Practice manager questionnaire. An online survey was sent to all practice managers, covering three principal areas: (1) their experience of using the tool (including the amount of time it took, the mechanics of using the online platform and the availability of data required), (2) their perceptions of its usefulness and how well it could be used to improve practice effectiveness and (3) contextual information about the practice including major changes to staffing and overall practice expenditure over the period.

Data from the interviews and open-ended question data from the practice manager questionnaire were coded and analysed using thematic analysis. Findings from all three strands were then collated to draw overall conclusions about the tool.

Stage 1a: ProMES-based workshops

• Phase 1: two large mixed workshops with staff and public (objectives) • Phase 2: six small workshops with homogeneous groups (indicators) • Phase 3: two large mixed workshops with staff and public

(contingencies)

Stage 1b: finalisation of tool

• Consensus exercise to agree details of measure, contingencies and relative weightings of objectives

• Development of online version of tool in preparation for practice use

Stage 2a: piloting of tool

• Use of tool by 50 general practices across multiple CCGs and regions • Six-month pilot period with feedback provided monthly

Stage 2b: evaluation of pilot • Evaluation of feasibility and acceptability

• Telephone interviews with a representative from each practice • Online survey of practice managers

• Analysis of data entered by practices

Stage 2: piloting and evaluation

Stage 1: development of measure

FIGURE 4 Overall summary of the research design.

OVERALL METHODS

NIHR Journals Library www.journalslibrary.nihr.ac.uk 30

Chapter 4

Stage 1: developing the measure