During Stage 1, you identified evaluation methods and drafted an outcome evaluation plan. At that time, you should have collected any necessary baseline data. The first step in Stage 4 is to review that plan to ensure it still fits your program. A number of factors will influence how your communication program’s outcomes should be evaluated, including the type of communication program, the communication objectives, budget, and timing. The outcome evaluation needs to capture intermediate outcomes and to measure the outcomes specified in the communication objectives. Doing so can allow you to show progress toward the objectives even if the objectives are not met. Consider the following questions to assess the Stage 1 outcome evaluation plan and to be sure the evaluation will give you the information you need:
EXAMPLES OF EFFECTIVENESS MEASURES FOR HEALTH COMMUNICATION PROGRAMS Knowledge
A public survey conducted before and after NCI’s 5 A Day campaign found that knowledge of the message (a person should eat 5 or more servings of fruits and vegetables each day for good health) increased by 27 percentage points.
Attitude
In 1988, the U.S. Surgeon General sent a pamphlet designed to influence attitudes on AIDS to every U.S. household. An evaluation conducted in Connecticut showed no change in attitude between residents who read the pamphlet and those who did not.
Behavior
The Pawtucket Heart Health Program evaluated a weight-loss awareness program conducted at worksites. More than 600 people enrolled, and they lost an average of 3.5 pounds each compared with their
preprogram weight.
• What are the communication objectives? What should the members of the intended audience think, feel, or do as
a result of the health communication plan in contrast to what they thought, felt, or did before? How can these changes be measured?
• How do you expect change to occur? Will it be slow or rapid? What measurable intermediate outcomes (steps toward the desired behavior) are likely to take place before the behavior change can occur? The behavior change map you created in Stage 1 should provide the answers to these questions.
• How long will the program last? What
kinds of changes can we expect in that time period (e.g., attitudinal, awareness, behavior, policy changes)? Sometimes, programs will not be in place long enough for objectives to be met when outcomes are measured (e.g., outcomes measured yearly over a 5-year program). To help ensure that you identify important indicators of change, decide which changes could reasonably occur from year to year.
• Which outcome evaluation methods can capture the scope of the change that is likely to occur? Many outcome evaluation
measures are relatively crude, which means that a large percentage of the intended audience (sometimes an unrealistically large percentage) must make a change before it can be measured. If this is the case, the
evaluation is said to “lack statistical power.” For example, a public survey of 1,000 people has a margin of error of about 3 percent. In other words, if 50 percent of the survey respondents said they engage in a particular behavior, in all likelihood somewhere between 47 percent and 53 percent of the population
represented by the respondents actually engages in the behavior. Therefore, you can conclude that a statistically significant change has occurred only if there is a change of 5 or more percentage points. It may be unreasonable to expect such a large change, and budgetary constraints may force you to measure outcomes by surveying the general population when your intended audience is only a small proportion of the population.
QUANTITATIVE VERSUS QUALITATIVE EVALUATION
Quantitative research is used to gather objective information by asking a large number of people a set of identical questions. Results are expressed in numerical terms (e.g., 35 percent are aware of X and 65 percent are not). If the respondents are a representative random sample, quantitative data can be used to draw conclusions about an intended audience as a whole. Quantitative research is useful for measuring the extent to which a knowledge set, attitude, or behavior is prevalent in an intended audience.
Qualitative research is used to gather reactions and impressions from small numbers of intended audience members, usually by engaging them in discussion. Results are
subjective and are not described numerically or used to make generalizations about the intended audience. Qualitative research is useful for understanding why people react the way they do and for understanding additional ideas, issues, and concerns.
Quantitative research methods are usually used for outcome evaluation because they provide the numerical data necessary to assess progress toward objectives. When
evaluating outcomes, qualitative research methods are used to help interpret quantitative data and shed light on why particular outcomes were (or were not) achieved. See the Communication Research Methods section for detailed explanations of quantitative and qualitative research methods and the circumstances under which you should use each.
ST
AGE 4
• Which aspects of the outcome evaluation plan best fit with your organization’s priorities? Only rarely does a
communication program have adequate resources to evaluate all activities. You may have to illustrate your program’s contribution to organizational priorities to ensure continued funding. If this is the case, it may be wise to evaluate those aspects most likely to contribute to the organization’s mission (assuming that those are also the ones most likely to result in measurable changes).