Despite the growing body of literature outlined in the previous section, there is relatively little evidence of
PPI’s impact and outcomes.99Several factors contribute to the weak evidence base for PPI. First, previous
evaluations have failed to use methodologies that take into account its complexity,100arising from PPI
being an interconnected, multifaceted social process, making it difficult to pinpoint specific impacts and
contributions.19Second, there is little systematic appraisal and reporting of PPI impact and outcomes,101
further constrained by the conventions of academic publishing.21Nevertheless, there is a limited number of
The systematic review by Oliver et al.102was the first to examine the evidence base on the impact of PPI on identifying and prioritising possible topics for research and development. They found detailed reports of 87 examples of PPI influencing the identification or prioritisation of research topics. While these reports highlighted the collective experience of research programmes in involving patients and the public, gaps
were identified in training and preparing lay people. It also highlighted multiple barriers to service users’
ideas influencing research agendas, and a lack of collaborative working between the research programmes and lay people to address the way forward in collective decision-making. In addition, the majority of included reports were descriptive accounts written by researchers rather than service user representatives.
Staley’s 200919review of the literature appraised papers published from 2007 within the INVOLVE
collection of articles, systematically searched electronic databases and drew on grey literature obtained
through networks via INVOLVE. A total of 89 articles (71 published papers and 18 from‘grey’ informally
published literature) were included in the review. While there was some evidence around the impact of PPI on influencing research questions, design, data collection, dissemination, implementation and research
ethics, the most substantial body of evidence was on PPI’s impact on recruitment. However, particular
gaps in evidence were found on PPI impact around grant-funding decisions, and on involvement in data analysis apart from qualitative methodology. There was some evidence of both positive and negative impacts of PPI on the public and researchers involved, and there were examples of positive impact for research participants in terms of PPI influencing a better research process, and impact on the wider
community. However, as with Oliver et al.’s review,102because of the nature of the evidence available,
inconsistent reporting and the context-dependent nature of PPI, it was not possible to carry out an in-depth analysis of different kinds of impact, or to judge the quality of the evidence and determine how generalisable the results were.
The Patient and Public Involvement in Research: Impact, Conceptualisation, Outcomes and Measurement
(PIRICOM) systematic review29aimed to examine the conceptualisation, measurement, impact and outcomes
of PPI in health and social care research. Ninety papers published between 1995 and 2009 were included. Data were extracted from 83 papers (the remaining seven were not written in a format suitable for data extraction but were deemed sufficiently important to include within the conceptualisation of PPI). Of the 83 papers, 2 were randomised controlled trials (RCTs), 52 qualitative studies, 15 case study methodology,
4 cross-sectional studies and 10 structured reviews. As in Staley’s review,19the papers provided more
evidence around the processes of PPI rather than impact and outcomes, and found scant evidence of PPI impact on funding decisions. Hence, while there appears to be a reasonably robust level of evidence around the‘architecture of PPI’, reporting of outcomes continues to be poor. Brett et al.29argue that most studies have attempted to conceptualise PPI mainly around context and process and have not sufficiently theoretically grasped the complexities of PPI. They did not find any evidence around systematic measurement of PPI outcomes, and furthermore found a lack of clarity on the differences between PPI impact and outcomes. Jagosh et al.103addressed the complexity of PPI by utilising a realist review design.104Focusing on
participatory research, which they define as the‘co-construction of research between researchers and
people affected by the issues under study . . . and/or decision-makers who apply research findings’,
they reviewed published papers and grey literature on 23 studies or programmes of research where both service users and researchers either identified or set the research questions, selected the methodology, collected or analysed data and used or disseminated the research findings. In each example they attempted to map out the relationship between the context (any condition that triggers or modifies the behaviour of a mechanism), mechanism (a generative force that leads to outcomes) and outcomes (intended and unintended). As part of their realist review they generated three hypotheses to test within
each example. Drawing on the concept of‘partnership synergy’,105their hypotheses were that partnerships
virtuous circle, so that conflict and disagreement between the stakeholders is not necessarily a bad thing, as it may build more robust synergies through successful negotiation and resolution. However, as with the
earlier reviews, Jagosh et al.103highlight the limited reporting of PPI and also suggest that negative
outcomes are less likely to be reported.
In a study published a year later, Shippee et al.101develop a conceptual framework (Figure 1) from
their systematic review. They critique previous reviews as focusing on only one part of their proposed framework, suggesting that their approach is more comprehensive and recommending its use to guide future reporting of PPI. From an original 202 papers, the review included 41, with the findings mapped against the framework. They conclude that the evidence of PPI impact on funding decisions continues to be weak, with a notable lack of studies identifying the impact of PPI in data collection and analysis. In contrast, there is a growing body of evidence on the impact of PPI in addressing ethical issues and enhancing the validity of findings. The authors recommend that research be undertaken to compare and systematically report different methods of PPI, and that there be more robust indexing of such studies.
Essential components of PPI
• Early PPI within research
• PPI should consist of people reflecting the community for whom the outcomes are relevant
• Scope for an active role
Patient and service user initiation
• Researchers see PPI as equal partners • All stakeholders know their respective roles • Colearning process • Reassessment and feedback
Building reciprocal relationships
Phases of research with PPI
• Agenda setting and funding
Preparatory phase
• Study design and procedures • Study recruitment • Data collection • Data analysis Execution phase • Dissemination • Implementation • Evaluation Translational phase
Conclusion
Embedding PPI in research continues within an overall policy framework which is built on specific
contemporary moral and consequentialist arguments. However, while the body of literature describing PPI processes and the architecture required to enable PPI continues to grow, it still provides only comparatively weak evidence for impact and outcomes. One reason for this is the difficulty that confused terminology or poor reporting poses for finding evidence of PPI impact in the literature. What evidence there is tends to be descriptive, largely qualitative, often retrospective case studies,19,29,106with only a few exceptional
prospective longitudinal case studies,15comparative multiproject evaluations107or studies employing
quantitative designs.108This reflects the significant methodological challenges in evaluating PPI as a complex
intervention, and the length of time before impact may become visible. Key messages from the cumulative reviews of evidence on PPI in research suggest that an appropriate methodological approach must be sensitive to context and processes and their influence on outcomes, that PPI within a research study may shift and develop with time so that there is a need for a longitudinal approach, and that different models of PPI need to be evaluated and compared through a standardised design. In the next chapter we present our methodology designed with these conclusions in mind.
Chapter 3 Research design and methods
A
s outlined in the previous chapter, PPI in research is a complex social phenomenon, and can act asan‘intervention’.109Providing a more robust evidence base for PPI policy requires moving beyond
observational evaluations of PPI.29,30More conventional approaches to empirical evaluation would seek to
examine whether or not programmes, interventions or innovations are effective using controlled trial designs to measure specified effects of each of a number of variables on outcomes. Although such designs are well suited to assess the effectiveness of clinical treatments, it has long been recognised110that such research designs are less able to identify how exactly variables combine to create outcomes111or to fully explain the processes through which programmes actually work. Furthermore, controlled trials are, by definition, able neither to examine the dynamic nature of those interventions which, will, of necessity, evolve over time nor to explore the effect and complexity of the wider social systems within which PPI takes place. For example, an experimental evaluation (RCT) which investigated the impact of PPI in developing participant information sheets (PISs) for a clinical trial108concluded that PPI had little or no impact on participants’ understanding of the trial or on recruitment and retention, and the authors were able only to speculate that a wide range of contextual factors and limited understanding of the mechanism of gaining consent were responsible for this negative outcome. Yet, as Staley et al. argue:
If an experimental evaluation is not designed in a way that considers the contextual factors and aspects of the mechanism that have the potential to influence impact, then it may produce inaccurate or over-simplified conclusions about when and how involvement makes a difference.
p. 632
Realist evaluation (RE) is one of several theory-based approaches to evaluation developed within the social sciences, which seeks to address dynamism and context, rather than control for them, to explain more adequately how and, importantly, under what circumstances programmes, interventions or innovations will
‘work’ in real-world complex systems, such as those related to health care/research systems.112Such social
science approaches highlight, specifically, how it can be the reasoning and actions of people (social actors),
rather than inherent characteristics of the interventions themselves, that make them work.113
In conclusion here, the evaluation of PPI in research necessitates a method of enquiry capable of capturing the interplay between outcomes and processes and the context in which PPI is conducted. As other authors have advocated in reviewing the evaluation evidence for PPI,32a realist evaluation approach114,115offered this research the capability of more fully investigating and understanding what type of PPI provided what kind of outcome in relation to different types of research and settings. We employed an overarching critical realist
framework116,117focused on the mechanisms embedded within PPI, to facilitate our understanding of
outcomes present or absent depending on how they were triggered, blocked or modified.118,119RE,115which
draws upon this perspective, was used to inform the research design.
We continue this chapter with a brief introduction to RE and its application in this study. We then explain the three-stage research design and methods employed to test the links between context, mechanisms and outcomes of PPI. We also introduce Normalisation Process Theory (NPT) as the candidate programme
theory to explain the‘work’ required to embed PPI as normal practice. In particular, the focus of NPT on
social action can help understand how and how far PPI may be‘embedded’ within and across