This chapter focuses on measuring application and business impact, two of the seven measures that may be used to report the results of a leadership development program in a leadership score-card. The other measures are indicators, satisfaction, and learning, which were covered in Chapter 3, and ROI and intangible benefits, which will be covered in following chapters.
Measuring the actual application and business impact of skills and knowledge is important because these steps play a critical role in the overall success or failure of a leadership development program. If newly acquired skills and knowledge are not applied effectively, there will be no change in the performance of the individual and the cor-responding business impact—and no benefit from the leadership development program. As discussed briefly earlier, the value of infor-mation increases as progress is made through the chain of impact from satisfaction (Level 1) to ROI (Level 5). Thus, information con-cerning application (Level 3) and business impact (Level 4) is more valuable to management than reaction/satisfaction (Level 1) and learning (Level 2). Measuring application and business impact also provides the leadership development function with the opportunity to identify the barriers and enablers to application.
Several data collection methods are available to measure applica-tion and business impact. The range of possibilities vary including the use of questionnaires, interviews, focus groups, and observation.
Also included are several specific methodologies, such as action plan-ning and performance contracting. These data collection methods are explored in this chapter.
80
Questionnaires
The following items represent a comprehensive list of question-naire content possibilities for capturing follow-up data (Harrell, 2001). Figure 4-1 presents a questionnaire used in a follow-up eval-uation of a program on leadership development. The evaleval-uation was designed to capture data for an ROI analysis, the primary method of data collection being this questionnaire. This example will be used
Measuring Application and Business Impact 81
Figure 4-1. Leadership development questionnaire.
Figure 4-1. (Continued)
Measuring Application and Business Impact 83
Figure 4-1. (Continued)
Figure 4-1. (Continued)
to illustrate many of the issues involving potential content items for questionnaire design with emphasis on application (Level 3) and Impact (Level 4).
Progress with objectives. Sometimes it is helpful to assess progress with the objectives in the follow-up evaluation as is illustrated in question 1 in Figure 4-1. While this issue is usually assessed during the program (because it is Level 1 data), it can be helpful to revisit the objectives after the participants have had an opportunity to apply what has been learned.
Action plan implementation. If an action plan is required in the program, the questionnaire should reference the plan and determine the extent to which it has been implemented. If the action plan requirement is very low key, perhaps only one question would be devoted to the follow-up on the action plan, as illustrated in ques-tion 2 in Figure 4-1. If the acques-tion plan is very comprehensive and contains an abundance of Level 3 and 4 data, then the questionnaire takes a secondary role and most of the data collection process will focus directly on the status of the completed action plan.
Measuring Application and Business Impact 85
Figure 4-1. (Continued)
Use of program materials and handouts. If participants are pro-vided with materials to use on the job, it may be helpful to deter-mine the extent to which these materials are used. This is particularly helpful when operating manuals, reference books, and job aids have been distributed and explained in the program and are expected to be used on the job. Question 3 in Figure 4-1 focuses on this issue.
Application of knowledge/skills. As shown in question 4 in Figure 4-1, it is helpful to measure application by determining the level of improvement in skills linked directly to the program. A more detailed variation of this question is to list each skill and indicate the fre-quency of use and the effectiveness of use of these skills. For many skills, it is important to experience frequent use quickly after acqui-sition so that the skills become internalized. In this example, ques-tion 5 addresses the skill frequency issue.
Changes with work. Sometimes it is helpful to determine what spe-cific activities or processes have changed about participants’ work as a result of the program. As question 6 in Figure 4-1 illustrates, the participant explores how the skill applications (listed previously) have actually changed work habits, processes, and output.
Improvements/accomplishments. Question 7 in Figure 4-1 begins a series of four business impact questions that are appropriate for most follow-up questionnaires. This question seeks specific accom-plishments and improvements linked directly to the program and focuses on specific measurable successes that can be identified easily by the participants. Because this question is an open-ended question, it can be helpful to provide examples that indicate the nature and range of responses requested. However, examples can also be con-straining in nature and may actually limit the responses.
Monetary impact. Perhaps the most difficult question (number 8 in Figure 4-1) asks participants to provide monetary values for the improvements identified in question 7. Only the first year improve-ment is sought. Participants are asked to specify net improveimprove-ments so that the actual monetary values will represent gains from the program. An important part of the question is the basis for the cal-culation, where participants specify the steps taken to develop the annual net value and the assumptions made in the analysis. It is very important for the basis to be completed with enough detail to under-stand the process.
Improvements linked with program. The next question in the impact series (question 9 in Figure 4-1) isolates the effects of the program. Participants indicate the percent of improvement that is related directly to the program. As an alternative, participants may
be provided with the various factors that have influenced the results and are asked to allocate the percentages to each factor.
Confidence level. To adjust for the uncertainty of data provided in questions 8 and 9, participants were asked to offer a level of confi-dence for the estimation, expressed as a percentage with a range of 0 to 100 percent, as shown in question 10 in Figure 4-1. This input allows participants to reflect their level of uncertainty with this process.
Investment perception. The value of the program, from the view-point of the participant, can be useful information. As illustrated in question 11 in Figure 4-1, participants are asked if they perceive this program to represent an appropriate investment. Another option for this question is to present the actual cost of the program so that par-ticipants can respond more accurately from the investment perspec-tive. It may be useful to express the cost as a per-participant cost.
Also, the question can be divided into two parts—one reflecting the investment of funds by the company and the other an investment in the participants time in the program.
Linkage with output measures. Sometimes it is helpful to deter-mine the degree to which the program has influenced certain output measures, as shown in question 12 in Figure 4-1. In some situations, a detailed analysis may reveal specifically which measures this program has influenced. However, when this issue is uncertain, it may be helpful to list the potential business performance measures influenced by the program and seek input from the participants. The question should be worded so that the frame of reference is for the time period after the program was conducted.
Barriers. A variety of barriers can influence the successful appli-cation of the skills and knowledge learned in the training program.
Question 13 in Figure 4-1 identifies these barriers. As an alternative, the perceived barriers are listed and participants check all that apply.
Still another variation is to list the barriers with a range of responses, indicating the extent to which the barrier inhibited results.
Enablers. Just as important as barriers are the enablers, those issues, events, or situations that enable the process to be applied suc-cessfully on the job. Question 14 provides an open-ended question for enablers. The same options are available with this question as in the question on barriers.
Management support. For most programs, management support is critical to the successful application of newly acquired skills. At least one question should be included on the degree of management support, such as 15. Sometimes this question is structured so that various descriptions of management support are detailed, and
par-Measuring Application and Business Impact 87
ticipants check the one that applies to their situation. This informa-tion is very beneficial to help remove or minimize barriers.
Other benefits. In most programs, additional benefits will begin to emerge, particularly in the intangible area. Participants should be asked to detail any benefits not presented elsewhere. In this example, question 16 shows the open-ended question for additional benefits.
Other solutions. The leadership development program may be only one of many potential solutions for improving performance. If the needs assessment is faulty or if there are alternative approaches to developing the desired skills or knowledge, other potential solu-tions could be more effective and achieve the same success. In ques-tion 17 the participant is asked to identify other soluques-tions that could have been effective in obtaining the same or similar results.
Target audience recommendations. Sometimes it is helpful to solicit input about the most appropriate target audience for this program.
In question 18, the participants are asked to indicate which groups of employees would benefit the most from attending this program.
Suggestions for improvement. As a final wrap-up question, par-ticipants are asked to provide suggestions for improving any part of the program or process. As illustrated in question 19, the open-ended structure is intended to solicit qualitative responses to be used to make improvements.
Improving the Response Rate for Questionnaires and Surveys Content items represent a wide range of potential issues to explore in a follow-up questionnaire or survey. Obviously, asking all of the questions could cause the response rate to be reduced considerably.
The challenge, therefore, is to tackle questionnaire design and admin-istration for the maximum response rate. This is a critical issue when the questionnaire is the primary data collection method and most of the evaluation hinges on questionnaire results. The following actions shown in Figure 4-2 can be taken to increase response rate. Collec-tively, these items help boost response rates of follow-up question-naires. Using all of these strategies can result in a 50 to 60 percent response rate, even with lengthy questionnaires that might take 45 minutes to complete.
Interviews
Another helpful collection method is the interview, although it is not used in evaluation as frequently as questionnaires. The
leadership development staff, the participant’s supervisor, or an outside third party can conduct interviews. Interviews can secure data not available in performance records or data difficult to obtain through written responses or observations (Kvale, 1996). Also, inter-views can uncover success stories that can be useful in communicat-ing evaluation results. Participants may be reluctant to describe their results in a questionnaire but will volunteer the information to a skillful interviewer who uses probing techniques. While the interview process uncovers reaction, learning, and impact, it is used primarily with application data. A major disadvantage of the interview is that it is time-consuming and requires interviewer preparation to ensure that the process is consistent.
Measuring Application and Business Impact 89
Figure 4-2. Increasing questionnaire response rates.
Types of Interviews
Interviews usually fall into two basic types: structured and unstructured. A structured interview is much like a questionnaire.
Specific questions are asked with little room to deviate from the desired responses. The primary advantages of the structured inter-view over the questionnaire are that the interinter-view process can ensure that the questionnaire is completed and the interviewer understands the responses supplied by the participant.
The unstructured interview allows for probing for additional information. This type of interview uses a few general questions, which can lead into more detailed information as important data are uncovered. The interviewer must be skilled in the probing process.
Interview Guidelines
The design issues and steps for interviews are similar to those of the questionnaire. A few key issues need emphasis.
Develop questions to be asked. After the type of interview is deter-mined, specific questions need to be developed. Questions should be brief, precise, and designed for easy response.
Try out the interview. The interview should be tested on a small number of participants. If possible, the interviews should be con-ducted as part of the trial run of the leadership development program. The responses should be analyzed and the interview revised, if necessary.
Prepare the interviewers. The interviewer should have the appro-priate level of core skills, including active listening, asking probing questions, and collecting and summarizing information.
Provide clear instructions to the participant. The participant should understand the purpose of the interview and know how the information will be used. Expectations, conditions, and rules of the interview should be discussed thoroughly. For example, the partici-pant should know if statements would be kept confidential.
Administer the interviews according to a scheduled plan. As with the other evaluation instruments, interviews need to be conducted according to a predetermined plan. The timing of the interview, the individual who conducts the interview, and the location of the inter-view are all issues that become relevant when developing a plan. For a large number of participants, a sampling plan may be necessary to save time and reduce the evaluation cost.