The third case study included here illustrates another critical aspect of evidence utilization, namely that, even in the presence of abundant evidence, co-production can fail if interdependencies are not properly managed. As in the case of the TIA project, data for this case study were derived from direct observations of project activities and meetings,
interviews, and document analysis. Details of the data sources are provided in Table 15 below.
Table 15. Summary of Data sources and Information about the CHII project Interviews 1 with Project Manager – 13 with other PCT staff
Project
participants/involved throughout the project
From PCT: PH director, project officer, project manager, PH consultant, public engagement, commissioners of various services (MH, joint commissioners, health improvement, etc), CEO, finance, PCT board, PCT executive team, procurement
Commercial organisation ( ‘ResearchCo’): Project manager & Director, researchers, Workstream
leads Meetings observed 6 meetings Public meeting minutes 9 board meetings
Public documents Press releases, project progress reports, strategic plans, Q&A project reports, project deliverables, governance structures, timetables, PCT forms
4.5.1 Background
The CHII project took off as an ambitious and expensive initiative (with a budget of about £10m over 3.5 years) that aimed at tackling the biggest preventable causes of ill health and death in the borough of PCT Z. The PCT was determined to tackle the main causes of morbidity, mortality and, most importantly health inequalities (as identified over many years in all their strategies) in an innovative way. The aspiration was that in-depth local research would “examine the behaviour and attitude of people across a diverse population” (study synopsis). Furthermore, research results would inform the development of bespoke and suitable to the local context health promotion and improvement interventions and services.
The PCT allocated a substantial amount of funds to undertake this project, which was delivered mainly with the support of a commercial partner (a FESC qualified organisation17). The procured partner – hereafter called
17
A policy issued by the Department of Health in February 2007 ‘for procuring External Support for Commissioners (FESC) will provide Primary Care Trusts with easy access to a framework of expert suppliers who can support them in undertaking
ResearchCo (a pseudonym) – was commissioned to design and carry out a sophisticated research study that, if successful, would constitute the basis for the delivery of a series of comprehensive health improvement services. The research involved designing, conducting and administering a residential survey as well as reviewing international evidence of health improvement interventions and evaluations; and was sponsored by the PCT Public Health (PH) Director. ResearchCo was also to carry out a health needs assessment, develop an engagement strategy and implementation of suitable customised health promotion interventions, and devise an intervention evaluation
framework.
ResearchCo worked closely with the PCT to deliver the required outputs for each workstream. For example, there was a lot of data sharing with the PCT for carrying out the needs analysis. There was also a lot of interaction with the Communications Department of the PCT with regard to e.g. the survey. Most significantly, frequent project meetings allowed PCT’s staff to provide necessary input into certain aspects of the project (e.g. information about interrelated initiatives leg by e.g. Local Authority) and monitor ResearchCo’s progress. In addition, 2 senior members of ResearchCo spent 2 days per week each at the PCT offices. Finally, the PCT at an executive team level made a special effort in involving various stakeholders (voluntary
organisations, local authority, etc.) and in making sure that the work was widely publicised, referenced and integrated with other major developments (see Table 15 above).
As far as the project’s outcomes were concerned, the initial research was deemed very successful. Thousands of questionnaires from local residents were returned and a number of qualitative interviews were also conducted. Towards the end of the research, and in the middle of the uncertainties following the announcement of major reforms in the NHS, PCT Z set out to assess ResearchCo’s proposals and business case for a set of health
promotion interventions that would follow up, and build upon, the results of the research. A series of meetings were held between ResearchCo and the PCT (including bot the project sponsor and other commissioners) in order to consider the commissioning of a comprehensive programme of health
improvement interventions (CHII).
ResearchCo, whose revenue stream derived from contracts, had a big stake in the CHII initiative. The end of the initial research project was
approaching, and so was the potential termination of their contract. A
their commissioning functions….The framework has been developed in response to the vision set out in Health Reform in England: update and commissioning framework (July 2006) for stronger and more effective commissioning, as a key element
successful business case would secure an important source of income over the coming months.
The changes in the commercial relationship corresponded to a shift in the interactions between ResearchCo and the PCT staff. For example,
ResearchCo delayed sharing their proposals with PCT staff; they insisted that the proposal would be packaged in a specific way. They also priced the new contract at more than £1.5m without providing a breakdown of the costs. The PH Director and project and contract manager were supportive of some elements of the proposals, yet ResearchCo staff was disinclined to provide financial details. This created tensions between PCT and
ResearchCo, as manifested in the following dialogue that took place at a meeting.
PCT project manager: Can we have a breakdown of the costs of your proposals? ResearchCo staff: That is not possible at the moment…
PCT project manager: We need to understand the financial assumption… when commissioners see the proposals, they will need to know the breakdown… because some of what is there [in the proposal] may be produced in house… how did you get to £1.5m (overall value of the ResearchCo proposal?
ResearchCo staff: Maybe I made an error there….
PCT project manager: Without numbers the business case doesn’t mean anything to commissioners). They will need a number attached to each element (of the proposal)… otherwise it doesn’t have a value! We need the detail of
information… We like the approach, we agree on that (content of proposals)… but not for that amount of money! You (ResearchCo) need to succinctly demonstrate value.
Another source of friction was the so-called ‘return on investment’ (ROI) of the proposals. Each business case submitted to the PCT investment panel should demonstrate ‘good’ ROI. In this instance, however, the calculation of ROI was not straightforward, as Sheila from ResearchCo tried to explain to the PH Director:
Sheila, ResearchCo: It is difficult to do a return on investment on these things [referring to health promotion proposals]. It is difficult to know what impact they will have…
PH Director: I don’t think ROI is hard… you just create a model… We’ve got the evidence (contribution of preventative causes to ill-health and death) from the needs assessment…
Sheila, ResearchCo: (Objecting kindly) When you put ROI in a contract, it is quite different… It is difficult to put in a contract how many people will actually stop smoking as a result of a particular intervention… Helen [in charge for
because if she doesn’t deliver, she will lose her job. That’s how it works in the commercial sector… (Sheila, ResearchCo)
PCT Project manager: (But) People (commissioners) will become disinterested (without a ROI)…
A number of important difficulties gradually emerged. On the one hand, the PCT wanted the business case proposal to be very detailed and demonstrate succinctly investment value and within a short time period (6 months). This was particularly important for commissioners in the current restrained financial climate. On the other hand, ResearchCo concluded that delivering specific, measureable health improvement targets would have been very difficult to hit within the required timeframes. This meant that they were disinclined to define concrete improvements in health outcomes as contractual deliverables.
When the final business case was submitted, the significance of these difficulties was exacerbated in light of the “whole new financial situation”. Commissioners, finance, and PH staff met and jointly discussed the merits of ResearchCo’s proposals, at the core of which was the use of media as a means to create engagement with specific patient groups at which the health promotion interventions were to be targeted. Some expressed their uncertainty about value for money (VFM). Others were very sceptical about ResearchCo’s intensive involvement in specifying the content of the
commissioning proposals and with the lack of transparency of how these proposals came about. Most were uncomfortable with the ‘lack of options’
(ResearchCo had provided only one set of proposals). Other staff with previous experience of using various media as engagement tools with hard to reach groups (e.g. people with learning difficulties), also felt that the recommended implementations were a ‘jump’ from the research. They understood the approach but were surprised by the high cost of the proposed initiative. They felt that they were being duped and expressed their anger about it. Collectively, PCT staff expressed dissatisfaction with the proposals, scepticism, frustration, guilt (because an exemplary project might not actually deliver anything substantial in terms of commissioning and strategic benefits), even cynicism (one PCT commissioner dubbed the proposals plainly as “an attack on our NHS!”).
The final decision on ResearchCo’s business case was discussed at an
important project board meeting attended by the PCT CEO, senior PCT staff and ResearchCo senior project members. Although ResearchCo were
pitching again their business case with the motto “these are evidence-based proposals”, they refused to modify the proposals in order to accommodate some of the PCT’s requirements for e.g. alternative delivery vehicles and greater number of options.
The outcome was that the PCT did not approve the ResearchCo’s business case. Differences were not reconciled. While the business case addressed ResearchCo’s needs it did not comply with the PCT requirements. The proposal was in fact developed as a complete, and in many ways sealed, package. This fitted ResearchCo’s capabilities and its pressure to generate income, but did not provide options or demonstrate value for money as required by the PCT. In effect, the all or nothing rhetorical strategy adopted here not only severely undermined the PCT’s commissioning aspirations for a comprehensive set of health improvement interventions, but also affected its ability to achieve its strategic objective to reduce health inequalities.
Figure 4. Chronology of the key events, activities and outcomes of the CHII project
4.5.2 EMERGING THEMES
Evidence in use
Since our interest lay in commissioning, we focused on the business case development and evaluation, and on the ways multiple kinds of evidence were used at that stage. Due to space limitations, we summarise our key findings in the following Table 16.
Table 16. Evidence in use during the CHII project
What evidence was used? How was evidence engaged? How was evidence mobilised?
Public Health Intelligence: Evidence of where inequalities exist and why Universal: nationally and scientifically created
Enabled public health experts to talk about the problem of health inequalities to non-experts and to communicate with reasons what could be done, i.e. focus on the key causes of preventable ill health and
invest in opportunities to tackle health inequalities; afforded justification of investment decision (PCT Z’s decision to procure ResearchCo); legitimation of the decision in the eyes of multiple
audiences (SHA, DoH, non-executive directors, LA)
In PH director’s report, in PCT strategies, in annual reports, in conversations at board meetings Local Health Needs
assessment & findings from local research into
residents’ lifestyles and
attitudes Local: scientifically produced and in context
By way of ‘reporting’ key findings, highlighting ‘high-level’ research
results through narrative means (e.g. ResearchCo rep reported that “the
interesting stuff that is coming out is that ‘young people are cross at older people’ regarding their drinking habits”), using quotes that
illustrate key findings, concentrating on certain results & fitting emerging research findings with ResearchCo proposals; e.g. those in greatest need, also wish to give up smoking, therefore we need such
and such interventions
In meeting presentations, in executive summary
reports.
Evidence related to health improvement and marketing interventions Universal: scientifically produced
In order to explore what population to target; claim the ‘evidence-
basedness’ of proposals; raise credibility of proposals through referring
to authoritative body of knowledge (e.g. ‘judgement heuristics’, ‘theory of interpersonal behaviour’, behavioural economics, stage-matched
model of behavioural change); claim rigorousness of proposals
Embedded in presentations and introduction to proposals. Evidence of effectiveness of proposed interventions, i.e. of causal relationship between interventions and measurable outcomes
Limited evidence was provided, even though PCT requested repeatedly; e.g. why the ‘co-creation of health promotion messaging’ was a suitable mechanism to e.g. help someone give up smoking1; generally hard to
quantify?
Through business case forms and at pitching meetings
Business case supporting evidence: strategic, economic, commercial, financial, project management case Local: produced in context
In order to raise argument for the benefits of investing in ResearchCo proposals; in order for the PCT to scrutinise the investment merits of the
proposals. The evidence for economic, financial and commercial case appeared to have been scant. Value was not succinctly demonstrated2.
The cost of the proposals was denounced as ‘disproportionate’, while
the proposed targeted population was deemed inadequate or even inappropriate to address health inequalities (i.e. targeting some cohorts
of people would not lead to a reduction in health inequalities)
Through Investment Business Case forms
and at pitching meetings
1
The major difficulty in proving the effectiveness of proposal implementation became evident when ResearchCo
attempted to articulate the potential outcomes and ‘expected’ improvement of their proposals in the business case form.
Despite the abundance of evidence produced and used throughout the life of the initiative, from a commissioning perspective the proposal was far from evidence-based. The effectiveness of the proposed interventions had not been proved, and the supporting evidence for the business case was regarded as unsubstantial. Although the failed business case could be
attributed to ResearchCo’s inability to produce relevant evidence, as well as wider institutional influences on PCT processes18, it could also be argued
18
At the start of the project, the ‘institutional environment’ was favourable to the innovation and risk that the PCT was taking. The PCT was ‘ticking’ a lot of the national policy boxes (e.g. World Class Commissioning, WCC); for example, they were ‘needs led’ and looking at innovation, engaging private sector partners etc. Then, in summer 2010 when the business case was about to be considered, there was a seismic change –first the news of a forthcoming election and financial austerity,
that the final decision to reject the business case was conditioned by the inability of both ResearchCo and the PCT to manage key interdependencies in the process.
Interdependences and emerging boundaries
ResearchCo and PCT staff had to deal with a wide range of
interdependences. Our analysis suggests that many of the challenges they faced were associated with boundaries, which were drawn within the
process – including, particularly, those defined by ResearchCo in the course of developing and pitching their business case. Table 17 summarises our key findings regarding the interdependencies experienced throughout this initiative.
Table 17. Interdependencies in the CHII initiative
Source of Interdependence Response to interdependence
Consequences
Acknowledged interdependencies
Project work related: project governance interdependence (e.g. who reports formally to whom for what, milestones),
contractual (defining conditions of contract), role interdependences (who is doing what, how and when),
expertise (assembling and integrating dispersed expertise from both ResearchCo and PCT), stakeholder
management Strict project management, e.g. setting relevant objectives and structures Significant Project progress (deliverables, achieving milestones etc.)
Commissioning related: Commissioning health improvement interventions (CHII) should be based on
strong evidence; aligned with key strategic priority
(tackling health inequalities); CHII depends on engagement with targeted groups
Proactive management: defining clear scope of
programme as evidence-based, proposals aimed to engage residents Project scope considered highly innovative & exemplary Unacknowledged interdependencies (boundaries)
Project work related: procedural (providing the detail of information required by investment forms), relational
(good relationships enable cooperation), rhetorical (e.g. convincing PCT commissioners about proposals through the creation of options), expertise (commissioning input
required for making business case more polished)
No management (e.g. ResearchCo did not develop rapport with PCT regarding proposal
benefits, PCT staff felt duped)
Distrust
Commissioning related: demonstration of ROI
(interdependence: investment merits determined through provision of specific measureable returns), proof of
effectiveness of interventions (interdependence: investment justification requires strong evidence of intervention effectiveness); temporality of the causal relationship between CHII and actual health outcome (health benefits can be yielded in long time periods)
Poor management
(despite PCT’s request,
ResearchCo failed to justify the selection of interventions, temporal interdependence unacknowledged) Decline of proposals
obsolete … PCT staff panicked, became risk averse, and didn’t even want to share the ‘good part’ (findings of their local research), in case they would be criticised and bear legitimacy losses.
Co-production of Commissioning Solutions
Commissioning the CHII programme was a two-party (client- consultant) collaboration. Whilst at some level co-production of certain outcomes was achieved e.g. of needs assessment reports, research findings reports, etc., co-production of the business case was severely compromised. As we highlighted earlier, ResearchCo refrained from engaging with
commissioners, and failed to take their perspective into account. As a result of ineffective co-production, the business case was eventually declined. While PCT Z intended to leverage expertise in the commercial sector, their search for clear value for money within a relatively short time period clashed with ResearchCo’s interest in securing a commercially successful contract. A series of action that followed these diverging set of needs and interests exacerbated the differences and eventually led to the final PCT decision to decline the business case proposal.
Table 18. Co-production during the CHII project Whose expertise was
leveraged?
On what issue? How was expertise leveraged and synthesised? When? ResearchCo Designing, developing, delivering four work
streams of a major research study
On the basis of formal contractual obligations, co-
location, etc. PCT Public Health Needs assessments, fit of proposals with
strategic objective, monitoring evidence production by ResearchCo
Monitoring ResearchCo’s drafts and deliverables.
PCT Commissioners Business case evaluation It was used to make a
conclusive investment decision. PCT communications team Health promotion and public engagement
proposals
Through meetings
Finance team Business case evaluation It was used to make a
© Queen’s Printer and Controller of HMSO 2012. This work was produced by Swan et el. under the terms of a commissioning contract issued by the Secretary of State for Health
Project 08/1808/244 111