BBVA

The concept of ongoing quality improvement is inherent in the science-informed approach to clinical practice. The evaluation skills needed to ensure that treatment programmes and services are delivered in line with best practice standards and, as a whole, reliably achieve clinically meaningful improvements in patient health and satisfaction, are as important to acquire for clinical psy- chologists as the therapeutic skills involved in the actual treatment of individual patients. The role of psychologists as healthcare providers in a competitive healthcare market (see also Chapter 13) makes it imperative that empirically- based outcome evaluation is not only conducted at the level of the individual patient, but is routinely extended to outcome evaluation at the aggregate level of the service provider or agency (see Figure 7.1).

Clinical psychologists by virtue of their combined training in research and clinical practice already have a solid grounding in research methodology and extensive experience in data management and analysis. This distinguishes them from many other healthcare providers and places them in a good position to make valuable contributions to addressing programme evaluation needs within their employment settings. However, programme evaluation involves more than competent data collection and analysis. In addition to data management skills, good evaluators need to develop good negotiation and communication skills (Owen & Rogers, 1999). Whereas research findings are instrumental in draw- ing conclusions that generalize to a broader context of a discipline or area of investigation, evaluation findings serve to inform decisions about specific aspects of a programme or policy within a local context. Hence, the conclusions that evaluators draw from quantitative and qualitative data will be influenced by the position held by various stakeholders. These may include consumers, policy makers, funding bodies, and staff and management of the respective agency (see Figure7.1). For example, evaluation data may be used to demonstrate that service provision achieves stated outcomes and is cost-effective, to account for the resources spent on developing and implementing a new service, to monitor

quality of care, to articulate the value of particular services to management and funding bodies, or to market services to consumers. In order to achieve these goals, the views and values of relevant stakeholders need to be considered. Good negotiation skills in the planning stage of an evaluation are essential to clarify and endorse the purpose of the evaluation, and to maximize the quality and utility of the data to be collected. Good communication skills are needed because evaluation data need to be translated into effective recommendations. Effective recommendations are those that are specific, realistic in scope, easily translated into action, and mindful of any constraints within the organizational environ- ment that might hinder their full implementation (Sonnichsen,1994).

Clinical psychology trainees typically experience two modes of skills training in programme evaluation: didactic teaching including practical experience in conducting small-scale evaluation projects, and modelling via exposure to and participation in ongoing programme evaluation activities within their training clinic and community placements. By embracing evaluation as a continuous

Figure 7.1. Programme evaluation extends routine evaluation of individual patient outcomes to evaluation of outcomes, procedures and policies at the level of the service setting

learning process of asking questions, reflecting on the answers to these questions, and modifying actions and strategies in light of those answers, trainees learn to be committed to a process of continual improvement that forms the basis of accountability and good practice. As shown in Figure7.1, members of the agency staff may be the direct beneficiaries of evaluation findings. To the extent that these findings are incorporated in continuing education and training of staff, programme evaluation adds to the wealth of clinical experience that practitioners bring to bear in treatment planning and implementation.

There is an enormous diversity in approaches to programme evaluation (Wholey et al., 1994), and the scope of many evaluation designs and purposes extends well beyond the local clinical context in which practitioners typically engage in programme evaluation. These more systemic evaluation projects are conducted by teams of professional evaluation contractors and are not the focus of this chapter. In this chapter, we introduce trainees to a few basic steps of programme evaluation that are common to the sort of evaluation projects that practitioners are likely to use in their respective local clinical setting on a routine basis. We will illustrate these steps with examples drawn from evaluation projects that clinical psychology trainees may encounter in their own clinical training setting.

Five basic steps of programme evaluation

1 Asking the right questions

The questions that form the impetus for a programme evaluation can be catego- rized according to their primary purpose. Owen and Rogers (1999) identify five conceptual categories. Proactive evaluation is conducted prior to the design and implementation of a treatment or programme. Questions addressed in pro- active evaluation might include: Is there a need for a particular treatment or programme? (e.g., should a women’s health centre offer a smoking cessation programme aimed at pregnant women? Is there a need for offering clinical psychology trainees practicum experience in rural and remote settings?). What does the relevant literature or professional experience tell us about the problems and benefits of introducing a particular service? Have there been previous attempts to address this need or problem? Are there external sources or agencies that could contribute expertise and solutions to problems in implementing a programme?

Clarificative evaluation examines the internal structure of a programme or policy. It clarifies how the programme’s elements and activities link to intended outcomes. Data will need to be collected that might address questions such as the following: What are the intended outcomes of the programme? What does the

programme do to achieve those outcomes? Is the rationale for certain aspects of the programme plausible? One example of clarificative evaluation that clinical psychology students may experience with respect to their own training pro- gramme is accreditation. Accreditation aims to certify that the structure, com- ponents, and procedural guidelines of a programme are of a standard that instils confidence that the programme can deliver what it intends to deliver.

Unlike clarificative evaluation, which is concerned with design and logic of a programme, interactive evaluation is concerned with implementation of a programme or its components. This form of evaluation is formative in nature and is particularly appropriate for the purpose of ongoing quality improvement. Questions asked in an interactive evaluation might include: Are there ways in which the delivery of services can be changed to make it more effective? Are therapists implementing agency practice guidelines? Do programme activities or innovative approaches make a difference? Are there changes in the type of patients and problems that present at the clinic? Are the skills taught in a training clinic up to scratch in meeting the demands of current and projected workplace requirements?

Monitoring evaluation aims to provide quantitative and qualitative informa- tion at regular intervals to gauge if performance indicators are in line with specified programme targets and implementation is carried out as intended. Questions asked during monitoring evaluation might include: How do patient outcome and satisfaction indices compare with the previous year? Are the resources available to therapists sufficient to meet current and projected trends in service delivery and patient needs? What is the average length of treatment provided by therapists? What is the rate at which clinical psychology trainees accumulate supervised client-contact hours? This type of evaluation often uses outcome monitoring data. Unlike programme evaluation, outcome monitoring itself is not explanatory, but simply generates routine reports of programme results, which are then available for periodic evaluation and interpretation. Thus, results-oriented monitoring generates the proof needed to satisfy the account- ability mandate (Affholter, 1994). In addition, outcome monitoring facilitates the early detection and correction of problems as well as the timely identification of opportunities for innovation and performance improvement (Affholter,

1994).

Finally, impact evaluation assesses the attainment of intended outcomes against specified criteria or outcome indicators. This category of evaluation is often summative in nature and may assist in decisions to scale back, terminate, continue, or expand certain services or programmes. Impact evaluation may also include an analysis of unintended programme outcomes and of the integrity of implementation. Questions addressed in impact

evaluation might include: Do patients receiving treatment in a training clinic achieve reliable and meaningful improvements in psychiatric symptoms and well-being? Are patients showing improvement in a timely and cost-effective manner? How long does it take graduates from a training programme in clinical psychology to find employment in their chosen field?

Once a set of critical evaluation questions has been determined that reflects the primary purpose and scope of the evaluation, the next step is to develop an evaluation plan.

2 Developing an evaluation plan

The second step of programme evaluation involves planning how to find answers to the critical questions agreed upon in the first step. Several issues need to be considered when negotiating an evaluation plan (Owen & Rogers, 1999). One consideration is who the recipients and users of the information are going to be. For example, if the question is how long it takes to find employment after graduation from a clinical psychology training programme, information will be primarily used by (a) current students from the programme for career planning, (b) future applicants to the programme for weighing the pros and cons of entering the programme, (c) potential employers for gauging the quality of graduates from the programme, and (d) programme directors for promoting the quality of programme graduates to employers and for advertising the strengths of the programme to prospective quality applicants.

A second consideration is what personnel and material resources are available to conduct the evaluation project. All evaluations are subject to resource and time constraints which determine the extent of information gathering, the complexity and sophistication of data management, and the range of dissemina- tion strategies. If the people who deliver a programme are also part of the evaluation team, then time and resources need to be made available for the evaluation tasks. These tasks should not present an additional burden on normal workload and should therefore be integrated within the routine demands of day-to-day programme activities.

A third consideration in developing an evaluation plan is selecting the data collection and management strategies that are most appropriate for each evalua- tion question. It is important to bear in mind that the level of data analytic complexity should not exceed the sophistication of the intended audience. Clarity and utility are paramount. To answer the question of how long it takes for graduates to find employment, a brief survey could be designed that examines not only time taken to secure the first job, but also provides further informative details such as number of job interviews relative to job offers, success in obtaining most preferred positions, duration of first contracts, starting income

level, time taken to progress to more senior or permanent positions and/or higher income levels. If data on prompt employment of graduates are not only of interest as an index of recent training success, but are also viewed as a performance target to be maintained in the future, data gathering may also require the simultaneous elicitation of critical feedback. Thus, the plan for data collection may include some open-ended forms of information gathering where respondents can identify areas for improvement in training, so that future graduates are kept abreast of evolving trends in knowledge and skills expected of their profession. In addition, the evaluation plan would articulate sampling strategies and may identify potential external resources that could aid in data collection. For example, a survey of recent psychology graduates could be mailed with the help of existing mailing lists kept by the university alumni services and could be included in their regular mailings at no cost to the evaluators. Finally, the data management component of the evaluation plan must specify how data will be processed and analyzed.

A fourth aspect in developing an evaluation plan concerns the strategies that will be used to disseminate the outcomes of the evaluation. This involves determining when and in what format reporting will take place, and what kind of results, conclusions, and recommendations will be included.

Finally, an evaluation plan must estimate the costs associated with carrying out the plan. The constraints imposed by the size of the budget and the amount of available resources have a direct impact on the timeline that can be set for achieving the various phases of the evaluation project. The timeline may also be constrained by any ethical issues that need to be addressed before and during the implementation of the evaluation. How these ethical issues will be handled need to be made explicit in the evaluation plan. After completion of the evaluation plan, the data gathering phase commences.

3 Collecting and analyzing data to produce findings

The third step of programme evaluation produces evaluation findings. These findings link the evaluation questions to answers that are then disseminated to relevant stakeholders (see Figure 7.2). The key tasks during the evidence gathering phase involve selecting and gaining access to the most appropriate sources of data, and then obtaining the data. Sources may include existing records and documents, as well as individuals who can either provide relevant information directly or are the gatekeepers of the information required. One of the most important tasks of data management is the maintenance of a reliable database. Clinical psychologists trained within a science-informed approach to clinical practice are usually very experienced in conducting and supervising the tasks involved in data reduction and analysis. Interpretation of the results of

the analyses must, of course, be grounded in the evidence, but in programme evaluation it is also important to ensure that the interpretation of evaluation findings reflects the diversity of viewpoints by different stakeholders. Are the conclusions based on a valid and balanced reflection of the evidence? Are there any limitations of the evaluation findings? Could these limitations have different implications for different stakeholders? It is essential to remember that evaluation conclusions must win the support of relevant stakeholder if they are to be utilized and lead to action.

4 Translating findings into recommendations for action

The fourth step of programme evaluation produces recommendations based on the judgements and interpretations derived from the evaluation findings. All recommendations have the purpose of influencing organizational decision- making. They may be used to justify decisions already made or, more typically, they inform and shape decisions about courses of action intended to bring about organizational change (Owen & Rogers, 1999). Whereas the first three steps of evaluation look backward and examine the status quo, the recommendations developed in the fourth step as part of the written evaluation report are designed to think forward. They are prescriptive and present solutions to problems, which in turn provide the impetus for organizational debate and action (Sonnichsen,

1994). As such, writing good recommendations is the most pivotal component

Figure 7.2. The collection, management, analysis and interpretation of data link the evaluation questions to answers for utilization by relevant stakeholders

of the final evaluation report if the evaluation effort is to bring about desirable change.

Effective recommendations are characterized by several basic qualities (Sonnichsen,1994). Foremost, recommendations must be delivered in a timely manner. Evaluation outcomes that are not available when they are needed are of little value to decision makers. Recommendations must also be realistic. Unless factors that might constrain the implementation of a recommendation are carefully considered, the recommendation may be viewed as impractical and hence is likely to be ignored. For the same reason, it is wise to avoid recommending changes that are so fundamental that they threaten values perceived by staff to be core aspects of the programme under evaluation. Such radical proposals for change are bound to meet with strong resistance and hence have little chance of being implemented successfully. In addition, care must be taken to direct each recommendation to the appropriate persons who are in the position to act on and oversee its implementation. Otherwise, recommenda- tions will collect dust and get bogged down in organizational inertia. Good recommendations are simple and specific. Each should focus on only one issue and make explicit what tasks are to be executed by whom to ensure its imple- mentation. Finally, the link between each recommendation and the empirical findings that underlie it must be clear. This will enhance the credibility of the recommendations and thereby their potential for acceptance and implementation.

The written recommendations can be seen as the ultimate product delivered by the programme evaluators. Because they are embedded in the final evaluation report, it is important that evaluators present their final report in a way that makes it easy for readers to understand the evidence behind a recommendation, the benefits of implementing it, and how to get there. If recommendations are entombed in a thick, unwieldy report, they are likely to remain unread, and hence cannot have an impact on ongoing quality improvement of service delivery. Therefore, they need to be displayed prominently in the final report. Sometimes adding page references to the relevant sections of the report can make it easier for the readers to find sections of interest. Although the specific content, structure and format of an evaluation report will depend on the target audiences and the guidelines imposed by funding bodies, most evaluation reports present upfront a brief section with an executive summary. This summary provides a brief overview of the evaluation aims, methods, and key findings, and highlights the recommendations that follow from these findings. The remainder of the report typically contains a more detailed introduction to the evaluation, a review of the literature, a description of the methodology, a succinct report of outcomes, and a discussion of the interpretations and judgements leading to

the recommendations. Further details are usually relegated to appendices in the form of tables or figures.

Even if great care has been invested in writing effective recommendations and presenting them in a way that they are quick to absorb and easy to understand by the stakeholders, this is not a guarantee that they will be adopted and lead to change. It may be desirable and possible, especially within local clinical settings where interactive forms of evaluations are often useful, for evaluators to be also actively involved in facilitating the transition from recommendations for action to the actual initiation and follow-up of the recommended changes.

5 Advocating and promoting change

Evaluation findings produce recommendations, but recommendations do not always influence organizational decision making. As Sonnichsen (1994) noted, ‘‘some evaluators labour under the delusion that elegant methodologies, eloquent reports, and scientific neutrality are sufficient qualities to ensure that evaluation results will be used’’ (p. 535). Non-use of recommendations may in part be due to the judgemental nature of evaluation. If recommendations for