Tineke Abma
This is a story about the politics of evaluation and subtle mechanisms of exclusion. A few years ago, an executive manager at Welterhof, a psychiatric hospital in the South of the Netherlands, approached me to conduct a responsive evaluation of a vocational rehabilitation project. The project was meant to assist and train (ex-) psychiatric patients in their search for a meaningful day activity or job. Project participants wanted to start on a small experimental scale in the garden and greenhouse. ‘Learning-by-doing’ was their motto, and they also reasoned that this ‘development along the way’ might profit from an evaluation. The purpose of the evaluation was not to assess the project on the basis of its effectiveness, but to motivate participants to reflect on their actions and to improve their practice. I considered the evaluation a wonderful opportunity to gather material for my PhD on responsive evaluation. A responsive approach to evaluation focuses on stakeholder issues and a dia-logue between stakeholders (Greene and Abma, 2001;
Guba and Lincoln, 1989). The approach requires a certain power balance to give all stakeholders equal opportunities as participants in the process. The challenge was how to conduct a responsive evaluation in a situation characterized by asymmetrical relation-ships.
Managers and staff
A creative therapist, who worked on a part-time basis for the project, assisted me in the evaluation. The first question we confronted was with whom to start. The assistant suggested that the people who were most directly involved in the development of the project
should be interviewed first, in this case the members of a specially formed task force. She had the feeling that some members were afraid that their work was not acknowledged while the manager was taking the credit for it. I took this observation seriously, because I did not want to ignore the invisible work of those who actually do the work. Being blind to these activities would be to succumb to the perils of managerialism.
The task force consisted of a heterogeneous group of practitioners with differing disciplinary back-grounds. Including them in a task force was uncom-mon in the hospital. Usually, projects were developed by a relatively homogeneous group of professional people. They would present their plans to the decision-makers and, what to do having been decided, the plans would be carried out by practitioners. The manager, who was relatively new in the hospital, considered this a very traditional leadership style.
Her motto was participation: ‘Involve people and share responsibility!’ She liked to emphasize that
‘involvement’ was part of her participatory manage-ment philosophy, and that this also corresponded with the rehabilitation philosophy. In her own words:
‘Not an expert-role towards the practitioner or an expert-role towards the patient, but jointly seeking the way.’
The vision of the manager differed remarkably from the experience of some of the members in the task force. We interviewed the staff and their stories suggested that they were not very happy with this new, participatory style of management. They liked being involved, but there were also signs that they did not like it. Some said, for example, that they wanted to be told what to do. Others interpreted the involvement as a delegation of work that needed to be carried out by them but that was not acknowl-edged. These paradoxical responses did not surprise us, because the message of the manager was also very paradoxical. On the one hand, people were invited to share responsibility; on the other hand several things were already predetermined. The manager, for example, preordained the planning: ‘The first year to experiment, the second year to improve, and third year to make a ‘‘go–no go’’ decision.’ The members accepted these time-constraints as hard and fast deadlines that could not be changed. This caused a feeling of panic especially among those who actually had to carry out the project: ‘We haven’t enough time to do the things that are needed to realize the quality we want.’
The imposition of the timeframe formed a hidden conflict between the manager and staff because the available time for the ‘experiment’ would have serious implications for their work. Compared to the hidden emotional response there was only slight overt resis-tance and this was only aired in private to a colleague of mine (Martin, 1992). The fact is that the staff did not encounter an overt conflict related to their self-definition. They did not consider themselves active ‘subjects’ who could influence the planning;
rather, they behaved as if they were merely passive
‘objects’ that had to adapt to the situations that confronted them.
As evaluators we decided to support the staff by suggesting to them that they should draw attention to their problem. Since my colleague was also a member of the task force she could join the little coalition.
Furthermore, we brought the subject up in one of the occasional meetings with the manager. We told the manager about the pressures of time that task force members felt under and asked her to adjust her deadlines to the rhythms of the people who were actually doing the project. Although this idea was at first contested, eventually the manager loosened her grip on the original plans. The negotiations over the planning and particularly the allocation of time to different phases were reopened.
Therapists and patients
The members of the task force were very eager to know what the patients thought of the project. ‘Are they satisfied with what we are doing?’ they asked. As evaluators we also found it important to take the patient perspective into account and wondered how we could make their silenced voices audible (Lincoln, 1993). We were very aware of the fact that madness was excluded from society the very moment that modern reason was born (Foucault, 1961/1984). How could we, as evaluators, let madness speak for itself?
How could we talk with the silenced in their (silenced) language? I was not satisfied with the procedural rules offered by Guba and Lincoln (1989: 17, 150). These excluded all those actors – young children, mentally handicapped, psychotics – who lack communicational skills, but who in fact only lack the skills that are required in a specific context created by evaluators to succeed with their chosen method. I thought that if one sticks to an ‘academic marketplace of ideas’ then metaphorical, playful and embodied aspects of every-day speech would be excluded. Furthermore, I was 1 2 T H E P R A C T I C E A N D P O L I T I C S O F S P O N S O R E D E V A L U A T I O N S
afraid that a rational debate would only reproduce the process of ‘othering’ instead of emancipating them.
In line with Schwandt’s (1994: 4) proposal for an
‘ethics of care’ that required ‘attention to particular others in actual contexts’ we began to participate in the activities of the group of patients who were taking part in the project. Initially we felt like voyeurs looking at people, but soon we began to forget about our role as ‘observer’. Both of us liked gardening and got caught up in the work. We developed a relation-ship with the patients and their facilitator, and learned about their activities, their lives and their concerns.
‘Patients,’ the facilitator (gardener) said, ‘are much more spontaneous in their reactions when they actually do something.’ We recognized this ourselves;
sitting or kneeling near someone’s body on the ground with your hands in the mud is less threatening than a face-to-face situation where one is interviewing the other. One could say that the question–answer method – even if the tone is nice – is always feeding dualism.
The facilitator was not the only one who was sensitive to the connection between knowledge and power. The managers, for example, remarked that
‘screening’ was not an appropriate word in the context of rehabilitation because it maintained the distance between professional and patient. Ironically, the manager was also the one who promoted the develop-ment and standardization of new methods and tech-niques to test and screen people. Most of the therapists embraced this proposal under the cloak that it would enhance the quality of their work. We showed them that observational methods also (or primarily) served another purpose: they established and maintained the professional power of therapists who are literally disciplining the bodies of those who are the subjects of these experts. A therapist tried to explain what this meant: ‘You create a different sort of relation. I mean, the relation therapist–patient is still there . . . and the patient does not need to be dependent . . . though . . . I have some expertise and that I find important too.’
There was still another even more subtle way by which therapists tried to hold power. We discovered this mechanism only later when we confronted them with our ideas about approaching patients.
After having developed a trusting relationship we decided to do a group interview with the patients.
This was not encouraged; members of the task force reminded us that we had to take all the precautions required by the law that protected the rights of patients. We felt in two minds about this. We could not bypass the ethical commission, but it was some-what patronizing that the patients could not decide for themselves whether to participate in the evalu-ation. While discussing the form of the group interview we suddenly came up with the idea of a picnic. This was less threatening than an individual interview. Moreover, we found it important to meet the patients in surroundings where they felt comfort-able – in this case, nature. Again the task group warned us with the sentence ‘They might become psychotic!’ We interpreted this caution as a resistance to share power, because asking patients what they think is indirectly an attack on the power-expertise of the professional. No longer would they (the profes-sionals) be the ones who knew what was best for the patients. Therefore we remained convinced of our idea and had a picnic. To everyone’s surprise the outcome surpassed all expectations. Patients were capable of expressing their wishes and their dissatis-faction once we adjusted to their world and language.
The task force members acknowledged they had underestimated the patients. The picnic as a different interaction with patients stimulated reflection and reopened fixated social relations between staff and patients.
In this story I have discussed the politics among project participants and how we as evaluators dealt with subtle mechanisms of exclusion as well as conflicts. Responsive evaluators have to be extra sensitive to power relations given the deliberate attempt to acknowledge plurality of interests and values and the genuine dialogue they want to facilitate (Abma et al., 2001; Koch, 2000; Wadsworth, 2001). In the case under consideration we as evaluators deliber-ately attempted to give voice to people and groups that are less powerful. We conducted in-depth inter-views that acknowledge the personal identity of people and created a safe environment where people felt comfortable to speak up. These stories and voices were amplified in the process. The discrepancy between what was said and actually done stimulated reflection on the side of both managers and thera-pists.
Annotated bibliography
Donaldson, S.I. and Scriven, M. (eds.) (2003) Evaluating Social Programs and Problems: Visions for the New Millenium. Mahwah, NJ: Lawrence Erlbaum Associates.
Some leading North American theorists’ views of evaluation and its social agenda including a transdisciplinary view, performance evaluation, empowerment evaluation, fourth-generation evaluation, a social activist approach, theory-driven evaluation and culturally sensitive evaluation.
Guba, E.G. and Lincoln, Y.S. (1989) Fourth-Generation Evaluation. Newbury Park, CA: Sage.
Although dated, this book has had a remarkable influence in shaping conceptions of the issues and practices of a social constructivist approach to evaluation.
House, E.R. (1993) Professional Evaluation: Social Impact and Political Consequences. Newbury Park, CA: Sage.
Explores evaluation as a modern social institution and its role in shaping society.
House, E.R. and Howe, K.R. (1999) Values in Evaluation and Social Research. Thousand Oaks, CA: Sage.
Explores the meaning and use of the concept of value in evaluation theory and practice. Advances a view of evaluation as wedded to deliberative democratic theory.
Kellaghan, T. and Stufflebeam, D.L. (eds) (2002) International Handbook of Educational Evaluation. Dordrecht, The Netherlands: Kluwer.
Coverage of key concepts, methods and areas of application in diverse national contexts.
Kushner, S. (2000) Personalizing Evaluation. London: Sage.
A unique way of looking at the relationship of people to programme in evaluation; instead of viewing programmes as the contexts in which participants’ experiences are read, it argues that participants’ experiences are the context in which social programmes and their significance are to be understood.
Patton, M.Q. (1997) Utilization-focused Evaluation. Thousand Oaks, CA: Sage.
Extensive explication of the theoretical and practical view that evaluations should be judged by their utility and actual use, where use includes both the application of evaluation findings and the value of participating in the process of evaluation. Also covers other forms of development evaluation.
Pawson, R. and Tilley, N. (1997) Realistic Evaluation. London: Sage.
A strong critique of current evaluation practice and an equally strong defence and explanation of evaluation methodology grounded in scientific realism.
Russon, C. and Russon, K. (eds) (2000) The Annotated Bibliography of International Program Evaluation.
Dordrecht, the Netherlands: Kluwer.
Annotated bibliographies on evaluation theory and practice for Africa, Asia, Australasia, Europe, Latin America, the Middle East and North America written by authors with extensive work experience in the respective regions.
Approximately 700 references are discussed.
Schwandt, T.A. (2002) Evaluation Practice Reconsidered. New York: Peter Lang.
A series of essays criticizing a narrow conception of evaluation as scientific methodology and arguing that evaluation practice ought to be redefined as a form of practical philosophy informed by the tradition of philosophical hermeneutics.
Stake, R.E. (2004) Standards-Based and Responsive Evaluation. Thousand Oaks, CA: Sage.
This book offers a balanced treatment of the choices evaluators face in framing programme evaluation in terms of standards and performance or in terms of activities, aspirations, and accomplishments of programme participants.
1 2 T H E P R A C T I C E A N D P O L I T I C S O F S P O N S O R E D E V A L U A T I O N S
Weiss, C.H. (1998) Evaluation: Methods for Studying Programs and Policies. 2nd edn. Upper Saddle River, NJ:
Prentice Hall.
A pragmatic approach to preparing for, designing and conducting evaluations in the complex political context of social programming.
Further references
Abma, T.A., Greene, J., Karlsson, O., Ryan, K., Schwandt, T.A. and Widdershoven, G. (2001) ‘Dialogue on dialogue’, Evaluation, 7(2): 164–80.
Chelmisky, E. and Shadish, W.R. (1997) Evaluation for the 21st Century. Thousand Oaks, CA: Sage.
Elliott, J. (2001) ‘Making evidence-based practice educational’, British Educational Research Journal, 27(5):
555–74.
Ferree, M.M., Gamson, W.A., Gerhards, J. and Rucht, D. (2002) ‘Four models of the public sphere in modern democracies’, Theory and Society, 31: 289–324.
Flyvbjerg, B. (2001) Making Social Science Matter. Cambridge: Cambridge University Press.
Forester, J. (1999) The Deliberative Practitioner. Cambridge, MA: MIT Press.
Foucault, M. (1984/1961) De geschiedenis van de waanzin (translation of Folie et déraison: histoire de la folie à l’age classique) (Published in English translation as Madness and Civilization: A History of Insanity in the Age of Reason). Amsterdam: Boom.
Greene, J.C and Abma, T.A. (eds) (2001) Responsive Evaluation. New Directions for Evaluation No. 92. San Francisco: Jossey-Bass.
Koch, T. (2000) ‘‘‘Having a say’’: negotiation in fourth generation evaluation’, Journal of Advanced Nursing, 31(1):
117–25.
Lincoln, Y.S. (1993) ‘I and Thou: method, voice, and roles in research with the silenced’, in D. McLaughlin and W. Tierney (eds), Naming Silenced Lives. New York: Routledge, pp. 29–47.
Martin, J. (1992) ‘The suppression of gender conflict’, in D.M. Kolb and J.M. Burtunek (eds), Hidden Conflict in Organizations, Uncovering Behind-the-scenes Disputes. Newbury Park, CA: Sage, pp. 165–86.
Power, M. (1997) The Audit Society: Rituals of Verification. Oxford: Oxford University Press.
Proctor, R.N. (1991) Value-Free Science? Purity and Power in Modern Knowledge. Cambridge, MA: Harvard University Press.
Schwandt, T. (1994) On Reconceptualizing Interpretive Educational Inquiry as a Normative Undertaking. Paper presented at the annual meeting of American Educational Research Association, New Orleans, LA.
Wadsworth, Y. (2001) ‘Becoming responsive – and some consequences for evaluation as dialogue across distance’, in J.C. Greene and T.A. Abma (eds), Responsive Evaluation, New Directions for Evaluation, No. 92.
San Francisco: Jossey-Bass, pp. 45–58.
Weiss, C.H. (1991) ‘Evaluation research in the political context: sixteen years and four administrations later’, in M.W. McLaughlin and D.C. Phillips (eds), Evaluation and Education: At Quarter Century. Chicago: University of Chicago Press, pp. 211–31.
PART IV
O B S E R V I N G , Q U E R Y I N G , I N T E R P R E T I N G
Introduction
This part of the book returns to the approaches and ideas presented in Part II and focuses more deeply on making meaning from human experience and social interactions in relation to different ways of under-standing the nature of being (ontology) and knowl-edge (epistemology). The chapters stem from philosophical foundations and are primarily con-cerned with observing people, interactions, discourses and activities in naturalistic settings. These data are then interpreted to present rich stories about people and the world they live in from their perspective rather than that of the researcher. The first chapter provides an introduction to different philosophical stances that can shape the researcher’s thinking and approach to data collection together with an introduc-tion to hermeneutics, a theory of interpretaintroduc-tion originally applied to biblical texts. The two chapters that follow describe methodological approaches which place emphasis on interpreting phenomena as experienced by research participants (or informants) in natural world settings. The chapter on observation provides an insight into how different ontological and epistemological starting points result in the collection of very different kinds of data, ranging from numeri-cal records of instances of particular behaviours to reflexive accounts that involve the researcher in an interpretive dialogue with data recorded holistically.
Finally, there is a chapter on discourse analysis, an analytical tool used to make sense of human com-munication through written and oral texts – interpret-ing and constructinterpret-ing meaninterpret-ing – gointerpret-ing beyond apparent surface meanings to uncover the connota-tions of power and emotion that lie beneath.
Observation is a valuable tool used in many research approaches, whether quantitative or qualitat-ive. But in Part IV the focus is mainly on immersion
in the field, gathering data intensively from a variety of sources in naturalistic – as opposed to experimen-tal – contexts, drawing on the researcher’s direct experience and often attempting to view participants’
experience from the inside (whether directly or indirectly). Interwoven with observation is querying or questioning, but not only questioning the partici-pants or people upon whom the research is focused.
A crucial aspect of the researcher as data gatherer is the capacity to question him/herself through a reflex-ive approach that takes account of the role of the self as a research instrument. How has the role of the researcher framed and shaped the interpretation through philosophical stances, insider/outsider ap-proaches and prior experiences for example? Interpre-tation also needs to take careful account of the context in which the data were collected or recorded and the effects of interactions. In discourse analysis for example, Bakhtin’s approach to the study of language is that words cannot be understood as transparent but rather as responsive – dialogic – and contextually embedded.
Once again, these chapters cross-refer to chapters in other parts of the book. Hermeneutics and phenomenology are philosophical approaches which have strongly influenced the practices of researchers using ethnography and case study (Part II). Ethical issues are of paramount importance when studying people, either directly through participation and shared experience, or indirectly through an indepen-dent, outsider’s view (Part II). There are also obvious cross-links to Parts VI and VII where observation reappears as an important method of recording
Once again, these chapters cross-refer to chapters in other parts of the book. Hermeneutics and phenomenology are philosophical approaches which have strongly influenced the practices of researchers using ethnography and case study (Part II). Ethical issues are of paramount importance when studying people, either directly through participation and shared experience, or indirectly through an indepen-dent, outsider’s view (Part II). There are also obvious cross-links to Parts VI and VII where observation reappears as an important method of recording