When asked how performance was measured, providers overwhelmingly indicated that it was “the numbers” or “ticking the box” that was important to the funder.
At the end of the day ... they’re numbers driven you know? How many people did we get through? You’ve got eight beds, are there eight people in them? Are those beds occupied 24/7, 13 months of the year? .... That’s what they’re driven
by....at the end of the day. MHP04B:3
It’s like, we look at the holistic picture in terms of mental health. And the way I see it, Ministry of Health, DHB, all they look at is numbers, numbers, numbers.
MHP07A:3
Basically we are required to report to them on a quarterly basis and they provide us with a reporting template and that really involves data collection of information that they want recorded. And so it’s just a matter of staff fill out
weekly .... stats sheets and then they are combined at the end of every month and then every three months. And they go in .... the quarterly report that goes to the
um, well goes to Dunedin for the DHB. MHP02A:2
Given that providers are sent a standard template which records counts in order to report back to the funder it is perhaps not surprising that the interviewed providers regarded “the numbers” as the crucial performance indicator for the funder. However, providers are also given an opportunity to raise other issues of concern in the narrative, or “Additional Information”, section of the performance monitoring reports. When asked about more descriptive means of reporting performance providers confirmed that they did fill out the Additional Information section of the reports, but that they rarely received feedback on these narrative reports.
Well the only thing that in any of the monitoring terms .... is they have an issues page where you can raise issues. Now whether anything is done with those issues, you’d really have to talk to the DHB. As a scenario we might say we haven’t reported this but .... look, 95% of our tangata whaiora are reporting housing issues. Now whether that is actually picked up at the DHB level and in
turn, do they talk to the Minister of Housing .... we don’t know. MHPO4B:4
Some providers also mentioned that audits were a type of performance measure, in that they were used to check on the quality of care or quality of service a provider might be delivering.
The internal processes are around quality control and quality improvement mechanisms. So we have things like .... clinical auditing of the client notes, we have case reviews ... and everyone takes part in those reviews, and that ensures that the care that we are giving to tangata whaiora is appropriate. We have peer supervision and .... then we have audits, like, we have, the Community
Nurse Clinician .... comes in and she does random audits of notes. MHP02C:3
However, while these audits were regarded as another performance measure, one provider noted that the audits were very much based upon “western” or mainstream indicators of quality:
I guess they come in and audit us, well they have us audited anyway, so yes there is a quality component to the agreement and its all mainstream. There is
nothing in there about kaupapa – it’s all mainstream. MHP03A:4
Adequacy of Performance Measures and Monitoring Returns
Providers were asked whether they thought the current performance measurement tools used by funders (i.e. the Performance Monitoring Returns comprising reporting units and the Additional Information section) adequately captured the extent of the work they did. A range of responses was received. Most providers agreed that the performance measures were inadequate - in particular the funders’ reliance on numbers as a de facto means of indicating improvements in client wellness.
If we talk about the ... reporting template, although they allow you to write .... whatever you like about certain issues there’s something in it that tells us that’s not the right template for reporting. I know my mental health team have ... a lot of difficulties with that, but you know at the end of the day, we just stay with it –
that’s what it is. MHP07A:4
I don’t think that they are actually getting the true picture of, like, what’s happening in the mental health area because they’re only really kind of seeking limited information and at the end of the day, its really just kind of output driven, how many people you have through and all that kind of stuff. Sometimes you get the feeling that that’s all they care about, how many get
spitted in and get churned out. MHP02A:5
This same respondent observed that the reporting mechanisms were “quite odd” particularly as it was not until the provider’s contract was renewed some three years later that a glossary of terms including definitions and specifications for the measures was ever sent to the provider.
... The first three years it was kind of a matter of us trying to figure out what it meant by what a group session was, or .... what was the difference between ... the number of people supported at the end of the period .... as opposed to the
numbers of people supported during the month. So we were kind of left in the
dark in that regard. MHP02A:3
The inadequacy of the monitoring returns was highlighted by another provider. In the Additional Information of their performance monitoring return for September 2002, the provider noted:
I believe the reporting system has a lot to be desired. There is nowhere in this report that I am able to record the total client contacts with clients during the
month. MHP03 Document E:10
One provider noted that neither the contracts nor the performance measures were actually intended to measure the amount or range of work undertaken by Māori mental health providers, stating:
Well they’re not designed to actually. They’re designed to capture the numbers .... They are designed to ensure that ... what we signed up for, we signed up for five things, we do five things. So they are designed to measure that .... We
report back what they want. We tick their boxes. MHP04B:5
Another provider commented that simply reporting the numbers would never allow the funder to have a true insight into the work of Māori mental health providers.
They don’t see the results. They can’t visually see, which is really difficult to put into words, to write about, they can not visually see what we see on a day to day basis. It’s just pure numbers [to the funder]. They don’t see the growth,
they don’t see the empowerment. MHP01A:3
Another provider, who agreed that simply reporting “quantitatively” does not demonstrate to the funder the extent of the work done, indicated that they were working on ways of reporting more qualitative information to the funder as part of their regular performance monitoring.
Showing them in terms of quality so rather than a quantitative measure, there’s a qualitative measure, that in fact, this is how many hours that we do spend with
our clients. So that a contact is not always about being 45 minutes, or being 60 minutes. That one contact may be three hours because of these reasons. So we’re trying to see how we can have our work measured qualitatively as
opposed to quantitatively. MHP05B
Respondents, when prompted, were able to identify specific areas where the current performance measures were inadequate. These included:
no provision to report the efforts of providers to make linkages and networks in the community (MHP02A:3);
no provision to record actual time spent with tangata whaiora and whānau, whether face to face or on the telephone (MHP02A:4, MHP02B:3 and MHP04B:3); and
nowhere to record holistic approaches to dealing with mental illness (MHP02C:5 and MHP03A:7).
One provider noted that the funder was too reliant upon numbers as an indicator of success and did not take into account the broader context in which Māori mental health providers worked. The respondent gave the example of a service where instead of ten clients in one reporting period, a provider only saw five. According to the funder, the provider has obviously failed in meeting their obligations to see the requisite number of clients; however the funder is unaware of any contextual information which may explain why a provider could only deal with five clients. It is for exactly these instances, the provider noted, that many providers keep a separate set of records (MHP07A:10).
The disadvantages are that there’s a lot of other work that we do as Māori that
isn’t accounted for so we slowly are building a database that highlights these things. Like, the number of hours we work with clients ... they don’t want to know that and ... you can only start counting a client when that client’s actually
accepted by the service ... But we as Māori spend a lot of time chasing people
up ... to even get to have that initial kanohi ki te kanohi … we’ll do that for two or three months. We have a very good paper trail that shows how much effort we make in getting hold of people. Recording of telephone calls, letters sent when people don’t arrive for appointment times, letters to the referrer to know