The responses of five experts to the Q and meta-evaluation summary findings identified influences associated with Aotearoa New Zealand’s historical, social, cultural and geographical context that may shape how evaluation is conceptualised and practised. This section summarises these influences and identifies their
potential impacts on evaluative reasoning.
In summary, New Zealand’s pragmatic heritage may contribute to evaluation being conceptualised as a practical, action-orientated undertaking. The pragmatic tradition may also contribute to greater value and importance being placed on the practice of evaluation rather than the theory on which such practice is based. This leads to evaluation practice that is described as being informal, less systematic and lacking rigour compared with countries where evaluation theory is given greater
emphasis. Such deficiencies may be compounded by a lack of external scrutiny of practice described by two experts.
New Zealand’s intimate society and relational way of being contribute to evaluation being conceptualised as a relational practice. This is reflected in the importance placed on stakeholder knowledge and input, and dialogic processes evident in the Q study. In a relational-based society the giving of negative feedback may not be socially acceptable, creating potential challenges for the way evaluators present evaluative conclusions/judgments. The Treaty of Waitangi, together with the influence of Māori and Pasifika cultures on the New Zealand way of being are described as creating greater awareness about the role of values in evaluation than may be the case in other countries in which the experts are based. Lastly, Māori evaluators are described as leading the international evaluation community in culturally responsive evaluation, and through their contribution to the cultural validity discourse as discussed in chapter two.
These conceptualisations of local evaluation practice raise questions about their potential impact on evaluative reasoning practice in the Aotearoa New Zealand public sector. While the following questions focus only on evaluative reasoning, many are equally relevant for other aspects of local evaluation practice.
i. What is required to engage the pragmatic evaluator with evaluation reasoning theory, and to understand the implications of such theory for practice?
ii. What is required to reconcile the pragmatic evaluator’s focus on the mechanics of doing evaluation with the cognitive deliberation and argumentation involved in evaluative reasoning?
iii. To what extent, and in what ways, does being an intimate, relational- based society impact on the way evaluators arrive at and present evaluative conclusions/judgments?
iv. To what extent, and in what ways, does the New Zealand propensity to avoid giving criticism and being unreceptive to criticism impact on the way evaluators arrive at and present evaluative conclusions/judgments? v. To what extent, and in what ways, do the relational and dialogic
features of local evaluation practice strengthen, or weaken evaluative reasoning?
vi. How do evaluators manage the tension between producing evaluative conclusions/judgments for accountability purposes and maintaining the mana of those being evaluated, particularly when evaluating Māori and Pasifika providers of government services?
vii. If external challenge and scrutiny of local evaluation practice are lacking, how does the evaluation community ensure the quality of public sector evaluative reasoning?
These questions suggest a range of potential impacts on evaluative reasoning practice. The purpose of identifying such questions is not to answer them, but rather to demonstrate how the application of evaluative reasoning theory may be mediated by context-related influences on evaluation practice. Responses to these questions present challenges to evaluation researchers in future.
8.9
Conclusion
This chapter has discussed the responses of three New Zealand-based and three international evaluation experts to a summary of findings from the Q study and meta-evaluation. These findings were endorsed by five experts who work or have worked in New Zealand as reflecting their experience of local evaluation practice. Although the experts responded to different aspects of the findings, underlying themes about the macro-level influences on evaluation practice were able to be identified. Such themes included New Zealand’s pragmatic propensities, the influence of Māori and Pasifika ways of being, our intimate society, and the importance placed on relationships. Such influences contribute to evaluation being conceptualised as both an action-orientated undertaking and a relational practice. Evaluation theory is described as being under-emphasised, leading to evaluation practice that is described as informal and lacking rigour. Such features of evaluation practice have potential impacts on evaluative reasoning practice in the Aotearoa New Zealand public sector. This is explored further in chapter 9.
PART D
BRINGING THE PERSPECTIVES TOGETHER
Chapter 9 brings together the perspectives on evaluative reasoning practice revealed from the Q study findings (chapter 6), meta-evaluation (chapter 7), and expert interviews (chapter 8) to address the three research questions: (i) how is evaluative reasoning understood and practised by evaluators working in, or commissioned by, the Aotearoa New Zealand public sector? (ii) how do contextual factors influence how evaluative reasoning is practised in the Aotearoa New Zealand public sector? and (iii) how can evaluative reasoning practice be strengthened in the public sector context? Chapter 10 (the conclusion) describes the study’s contribution to knowledge and identifies areas for further research.CHAPTER 9
EVALUATIVE REASONING IN
THE AOTEAROA NEW ZEALAND
PUBLIC SECTOR
9.1
Introduction
As described in chapter one, reading the literature about evaluation quality led me to a hunch: evaluative reasoning contributes to quality evaluation. This study set out to explore the conceptualisation and practice of evaluative reasoning in the context of public sector evaluation in Aotearoa New Zealand, and to identify how evaluative reasoning practice could be strengthened. A multiple method research design was used to generate diverse understandings on the topic aimed at creating potential opportunities for abduction. The outputs of abductive inquiry are
“ . . . plausible hypotheses, thereby opening up the space for others (or yourself) to find something else of interest that raises a new curiosity” (Wolf, Peace & Brown, 2015, n.p.). In section 9.4 of this chapter I use the concept of conjecture to present abductively-derived ideas arising from the study findings. I chose this word after reading Timmermans and Tavory’s (2012) reference to abduction as “the most conjectural” (p.171) of the three logics of deduction, induction and abduction. Used as a verb, conjecture means “to infer or arrive at (an opinion, conclusion) from incomplete evidence” (Collins, 2014), or “to conclude, infer, or judge from appearance or probability” (Oxford, 1991). While none of the thinkers in the field of abduction use the concept of conjecture in the way I propose, it seems to me there needs to be a term, perhaps even a sensitising concept (Patton, 2002a), through which such abductive findings can be framed.
The chapter begins with an explanation of how the key findings from the Q study, meta-evaluation, and expert interviews were examined together. Using inductive analysis, sections 9.2 and 9.3 compare the findings from the Q study, meta-
evaluation and expert interviews to answer the two descriptive research questions: (i) how is evaluative reasoning understood and practised by evaluators working in, or commissioned by the Aotearoa New Zealand public sector? and (ii) how do contextual factors influence how evaluative reasoning is practised in the Aotearoa New Zealand public sector? The concept of the evaluation imaginary (Dahler- Larsen, 2012; Schwandt, 2009b) is used to address this second question. Using abductive analysis, section 9.4 addresses the third research question: how can evaluative reasoning practice be strengthened in the public sector context? The final section (section 9.5) draws on the study findings to explicate the connection between evaluative reasoning and evaluation quality.