The probing value of any study needs to be considered, relative to its limitations (Speroff, 2004).
KPP does not directly measure clinical outcomes, rather it measures process: whether or not services are delivered, e.g. the number of patients with relapse prevention plans. The social indicators that KPP uses, such as work, are of major significance in any person’s life.
Multiple base line designs minimise weaknesses in the research design (Shadish et al., 1991). Ambiguity relating to causality of the KPP method has been minimised, by ensuring implementation fidelity and by ensuring that significant secondary base line data is available for analysis.
Diffusion obscures the true change effects in the treatment group (Cook & Campbell, 1979). Diffusion was recognised as a threat to the validity to this research, as a result of the KPP DHBs communicating within and between each other — and because the Ministry of Health introduced accountability requirements, as a result of some early findings in this research.
The stakeholder survey was only conducted in three DHBs and responded to by a total of 22 stakeholders, making the validity and reliability of the findings of this survey somewhat compromised. Patient advisors and some long–term patients were involved in the development of KPP. Only patient advisors were respondents to the stakeholder survey and that also limits the value of the survey findings. However, it is actual long–term patient data that comprises the KPP data set.
There is a possibility that the “Hawethorne effect” was in play during this research, in that DHBs were being closely observed over the course of the research. However, based on recent reporting to the Ministry of Health, the DHBs involved in this research seem to have continued with KPP approach post this research.
The service profile data, used in the quasi–experimental pre–test post–test service analysis, is drawn from MHINC, the national mental health data collection. MHINC is a ‘live’ system, so data can change depending on the date the data was extracted.
‘History’ is an event that takes place, in addition to the measure under observation. An example of a relevant history event was the introduction of the ‘Strengths’ treatment model into South Canterbury (SouthCant) DHB. The validity threat is minimised by the multiple action research cycles over multiple sites. The New Zealand health system design and mental health policy were stable over the period of this research.
Selection may impact on the findings, because KPP DHBs are inherently different to the non-KPP DHBs. This validity threat, however, is not seen as significant and it is discussed in Chapter Six.
The generalisability of the results, to New Zealand’s mental health services, seems unlikely to be questioned, since the research has been conducted in a wide range of DHBs including: rural; urban; large; small; and those DHBs with high and low funding and varying ethnicity mix.
I was able to gather and check KPP data in real time, because I was both the joint KPP Project Manager and I was also employed by the Ministry of Health. This helped to make timely and accurate data available for the research.
The Ministry of Health only uses the SAS statistical package and given I am not trained in SAS programming, the statistical analysis of the data was undertaken by a Ministry Biostatistician.
Starbuck (2004) contended that natural experiments are the research approach of choice and they occur when exogenous events displace social systems, from their normal equilibrium. In these situations, one can see some of the systems adaptive and reactive capabilities and this opens the possibility of discovering why the
equilibrium exists. KPP and this evaluation equate with a natural type of experimental approach.
5.8 Conclusion
Undertaking an implementation evaluation ensures that a distinction is made between programme theory failure and implementation failure. Emery’s socio- ecological action research provides an excellent method to determine the outcome of KPP — and to develop a theory. The research is data rich and it has considerable base line data, which enables significant testing for other factors that may be causing the result. Whilst individually, the validity of each of the research methods and their data sources can be questioned, when the results are synthesised, they represent a robust evaluation. The outcome results are recorded in Chapter Seven, whilst the KPP implementation results follow immediately in Chapter Six.
Chapter Six
KPP Implementation Fidelity
Undertaking an implementation evaluation enables a distinction to be made between implementation failure — and programme theory failure. Eight of New Zealand’s 21 DHBs were involved in this research. This chapter details how KPP implementation fidelity was achieved by those eight DHBs and it begins with an implementation overview and a map of the DHB locations. This is followed by a profile and implementation summary, for each DHB. Any issues peculiar to a DHB, which may have influenced the research findings, are also highlighted in the implementation summary. The results of the KPP stakeholder implementation survey are then summarised and discussed. Prior to the conclusion, I provide an overall implementation summary and I discuss possible reasons why KPP was adopted by some DHBs — and not others.