Interés técnico

Again, we do not intend to provide a detailed description of the process of conducting a realist synthesis; this can be found elsewhere.82_{However, here we do offer a blueprint, so that readers who are not familiar}

with the methodology can make sense of how we operationalised our synthesis. It also makes explicit our understanding of how a realist synthesisoughtto be conducted, which can be subject to scrutiny in judging the rigour and quality of the review.

Realist synthesis is an iterative review methodology, consisting of five main steps. For ease, these are described sequentially, but, in practice, there is considerable movement back and forth between different steps. Furthermore, a number of these steps, for example searching, quality appraisal, data abstraction and synthesis, are integrated rather than conducted separately.

Step 1: searching for and identifying programme theories

The basic unit of analysis in realist synthesis is not the intervention, but the ideas and assumptions or programme theories that underpin it. Thus, the starting point of realist synthesis is to search for and

catalogue the different ideas and assumptions about how interventions are supposed to work. Initially, these programme theories focus on practitioner, policy-maker and participant ideas and assumptions about how the intervention is intended to work (or not). These may specify the sequences of steps required to deliver the intervention, and the organisational and social processes required in order for the intervention to achieve its intermediate and final outcomes: that is, an‘implementation chain’. They may also identify potential blockages in this process, as well as potential unintended consequences. They often contain ideas about the different reactions or responses that participants may have to an intervention (mechanisms) that will determine whether or not the intervention is successful (outcome). They may also include ideas about the circumstances (or context) that determine the kind of reactions participants may have to an intervention, and the blockages that may occur, which thus influence the impact of the intervention.

These ideas often remain tacit and unexpressed in empirical evaluations of the intervention, which frequently assume shared knowledge regarding how the intervention is intended to work or consider that knowing how the intervention works is not important: the task is simply to know whether or not it works. Therefore, unearthing and cataloguing these ideas is best achieved through searching and analysing policy documents, position pieces, comments, letters, editorials, critical pieces and websites or blogs that express and debate these tacit assumptions and explain how the intervention in question is intended to work. For some interventions, it can be a useful exercise to deconstruct empirical investigations or policy documents of a given intervention to surface the implicit assumptions underpinning the design of the intervention itself.62

This often requires a considerable amount of‘detective’work and reflection on the part of the researcher. We explain how we searched for the programme theories underlying PROMs feedback in the more detailed description of methodology inChapters 2and6.

Step 2: focusing the review and selecting programme theories

Inevitably, the search for programme theories results in the identification of many different ideas about how the intervention is supposed to work, and its potential blockages and unintended consequences. It is not possible to review all of these, and the next stage of realist synthesis involves a process of (1) identifying common mechanisms or issues across the different programme theories, and (2) prioritising which set of programme theories to review.

The first is an important initial step in developing‘middle range’theory, which allows transferable lessons to be made. It requires the researcher to think‘what are these programme theories an example of?’and ‘how do these programme theories relate to more formal or abstract theories?’. Thus, it represents a process of moving up and down a ladder of abstraction, from practitioners’ideas about how a specific intervention works, to more abstract ideas about how the family of interventions which share that programme theory are expected to work. This plays an important part in defining the boundaries of the

review, as it serves to identify other interventions that also share the same programme theory, evaluations of which might therefore be included in the theory-testing phase of the review.

The second is a process of narrowing and deciding which of these theories we might focus on. There are no set criteria to govern these decisions,82_{but they can focus on:}

l which aspects of the programme theory stakeholders and practitioners consider most important or would like to be answered

l understanding how and why one particular section of the implementation chain works or becomes‘blocked’

l considering how the same programme theory fares in different contexts

l adjudicating between rival ideas about the mechanisms through which an intervention is intended to work.

These decisions then form the framework for the review. However, it must be recognised that this process is iterative. Inevitably, the process of testing one programme theory uncovers a number of‘sub’or‘mini’ theories within the review. Furthermore, the review is also likely to focus on a smaller number of main theories. Therefore, defining and redefining the boundaries of the review is an ongoing and iterative process. We explain how we identified relevant abstract theories and how we narrowed down the focus of our review inChapter 2for review 1 andChapter 6for review 2.

Step 3: searching for empirical evidence

The programme theories to be tested provide the backbone of the review, and determine the search strategy and decisions about study inclusion into the review in order to test and refine these theories. The next stage of the review thus involves an evidence search to identify primary studies that will provide empirical tests of each component of the theory. This involves electronic database searches, as well as forwards and backwards citation tracking. Searching and synthesis are interwoven, and, as the synthesis progresses, the emergence of new subtheories or mini theories often requires further iterative searches to identify empirical evidence to test them. Furthermore, the review is also likely to focus on a smaller number of main theories as the synthesis progresses. InChapters 2and6, we describe in some detail the processes we used to identify the empirical evidence on which this review is based.

Step 4: quality appraisal and data extraction

These are combined in realist synthesis. Different programme theories require substantiation in divergent bodies of evidence. Hypotheses about the optimal contexts for the utilisation of PROMs data are tested by comparing the outcomes of experimental studies in different settings, claims about the reactions of different recipients of PROMs data are tested using qualitative data, etc. Studies (or parts thereof) are included in the study depending on their relevance to the programme theory being tested.

Quality appraisal is conducted throughout the review process, and goes beyond the traditional approach that focuses on only the methodological quality of studies.87_{In realist synthesis, the assessment of study rigour}

occurs alongside an assessment of the relevance of the study, and occurs throughout the process of synthesis. Quality appraisal is done on a case-by-case basis, as appropriate to the method utilised in the original study. Both qualitative and quantitative data are compiled. In addition, the inferences and conclusions drawn by the authors of the studies are also extracted as data in realist synthesis, as they often permit the identification of subtheories that can then be further tested with empirical evidence. Different fragments of evidence are sought and utilised from each study. Each fragment of evidence is appraised, as it is extracted, for its relevance to theory testing and the rigour with which it has been produced.87_{In many instances, only a subset of findings}

from each study that relate specifically to the theory being tested are included in the synthesis. Therefore, quality appraisal relates specifically to the validity of the causal claims made in these subset of findings, rather than the study as a whole. Trust in these causal claims is also enhanced by the accumulation of evidence from a number of different studies, which provides further lateral support for the theory being tested, discussed in more detail in the following section. Finally, quality appraisal is integrated into the synthesis narrative, rather than reported separately.

Step 5: synthesis

The goal of realist synthesis is to refine our understanding of how the programme works and the conditions and caveats that influence its success, rather than offering a verdict, descriptive summary or mean effect calculation on an intervention or family of programmes. Synthesis takes several forms. At its most basic, realist synthesis involves building‘lateral support’for a theory by bringing together information from different primary studies and different study types to explain why a pattern of outcomes may occur. Another form of synthesis, particularly useful when there is disagreement on the merits of an intervention, is to‘adjudicate’ between the contending positions. This is not a matter of providing evidence to declare a certain standpoint correct and another invalid. Rather, adjudication assists in understanding the respects in which a particular programme theory holds and those in which it does not. Finally, the main form of synthesis is known as ‘contingency building’. All PROMs feedback programmes make assumptions that they will work under implementation conditions A, B, C, applied in contexts P, Q, R. The purpose of the review is to refine many such hypotheses, enabling us to say that, more probably, A, C, D, E and P, Q, S are the vital ingredients. InChapter 2, we will provide short examples of how we carried out our synthesis.