In the context of the effects-metrics framework promoted in the ELO Effects, Metrics, and Risk Workshop, various actions are taken to achieve the intended NCBC effects (outcomes), which—in turn—achieve the overall course objectives. Actions can be evaluated by measures of performance (MOPs), which are important for course design and quality. However, a well-designed course may still not achieve the appropriate course effects, which must be evaluated separately by measures of effects (MOEs). Both measures are necessary, but it is only through MOEs that we can see the impact of a learning experience and be assured that the Navy’s
investment in the NCBC is justified. The current ELO evaluation system is designed to collect MOPs but not MOEs. Information about the effects of the NCBC is seen only anecdotally when comments are volunteered by participants.
We can begin by looking at the current actions we take to produce course effects and ensure that we have consistent MOPs in place.
Actions
The actions taken in the NCBC to achieve the effects and, therefore, the objective of the course are, to some extent, implicit. Making them explicit results in the following list:
1. All module content is relevant and/useful to the current and/or future jobs of participants.
2. Presenters are experts in their fields, employ a professional speaking style, are able to engage participants with the topic, and make Navy applications clear though either examples or class discussion. 3. The 360-degree feedback coaches are knowledgeable about
organizational, leadership, and interpersonal issues; they provide constructive feedback, and provide excellent guidance for participants’ development.
5. Technical support for learning sessions is professionally delivered. 6. Communications prior to arrival at the NCBC, including all information
related to 360-degree feedback, are clear.
7. Participant lodging is comfortable and problem free. 8. The staff is responsive and provides excellent support. 9. Meals and snacks are varied and high quality.
10. The course facilities are conducive to a good learning environment. We can now turn to the means we use to measure these actions.
Analysis of Current Measures of Performance
The current evaluation devices and a mapping to the actions they measure are shown in Table 1.
Table 1. Current NCBC Measures of Performance Individual Session Feedback Working Group Feedback Overall Course Feedback 360 Coach Feedback End-of- course E-mail Action 1 X X X X 2 X X X 3 X X 4 X 5 X 6 X 7 X 8 X 9 X X 10 X
The table shows that all actions are measured in at least one evaluation. A review of the evaluation devices and the data from the past NCBCs produced by these forms suggests that the following revisions need to be made:
1. The data from the “Individual Session Feedback” evaluations have been very useful for speaker feedback and curriculum decisions. The average of the scores on this five-point-scale-form are typically high— in the 4.4. to 4.5 range for relevance and for the speaker, respectively. However, there are items from the form that are not useful (because they are not connected to any of the actions, nor do they seem
necessary). These can be eliminated. Additional data on these topics are generated anecdotally on the “Overall Course Feedback” and “End-of-Course E-mail.” A proposed revision to this form is shown in Appendix B. To get the maximum benefit from this evaluation, all presenters need to be encouraged to make a clear statement of their objectives.
2. The reactions to the projects documented in the “Working Group Feedback” are often a function of how the project is introduced, and are typically not very positive. Since we will likely continue to do the projects, we should consider eliminating this process to help cut down on survey fatigue.
3. The “Overall Course Feedback” form provides an important check on the support aspects of the course, as well as an opportunity for participants to comment on the intangible benefits. For example, a recent comment was, “I feel honored and humbled that the Navy
thought enough of me to invest in giving me this experience. I will do my utmost to ensure that the Navy gets an exponential return on this investment.” Additionally, the ratings for content and total experience are typically very high (e.g., 4.7 and 4.8 for content and total
experience, respectively). These are higher than the averages
produced on the evaluations of each individual module and may reflect the perspective of the bigger picture beyond modules, and may even include a feeling of goodwill resulting from the intangible benefits. As shown in Table 1, this form serves a variety of purposes and should be left intact.
4. The “360 Feedback and Coaching Evaluation” is the only real source of information about the coaching experience available and should be left intact.
5. The “End-of-course E-mail” also provides an opportunity for
participants to comment on intangible benefits and has proven to be quite useful for feedback on course content and the total experience. More importantly, a review of these data has served as a record of the “big nuggets.” It has also served as input into the design of a survey three to six months after the course (discussed in the next section of this report). Last, but also important, when this e-mail is sent via the
chain of command, it generates positive publicity for the program. This evaluation should be left intact.
Measures of Effectiveness
With actions and their corresponding MOPs clearly defined, the researchers can turn to the question of, “Are the actions resulting in the desired effects, which, in turn, will meet the course objective?”
The course impact can be measured through new survey or interview questions that correspond to our statement of effects and that are administered three to six months after the course. The researchers created a draft list of questions based on all of the data described above and pilot tested it on another (different from those used in the initial interviews) 20 NCBC graduates. The pilot test was conducted in two phases.
The first phase of the pilot test used the draft list of questions as a survey, which the researchers used as a prompt for a protocol analysis. In other words, we asked five graduates to think and answer the questions aloud as they went through the survey. Such analysis is typically used to ensure clear wording of the questions. What we quickly discovered was that these questions, as used in a survey, did not produce the kind of data useful to demonstrating the course outcomes. For
example, an answer of “strongly agree” that “I have made changes to my leadership style based on the 360-degree feedback coaching” does not begin to address the impact of the 360. It is only with the story behind that answer that we can start to see the value of the 360. Further, it is clear that even with the opportunity for open- ended responses, graduates were unlikely to spend a lot of time writing out the full “story” of the impact of the NCBC.
The second phase of the pilot test used the questions for interviews. Ten of the interviews were conducted face-to-face, and five were completed by phone. The face-to-face interviews were completed in graduates’ work spaces by the researcher who also facilitates the course. These two details seemed important to the success
of the testing. Graduates seemed pleased to be visited to have their opinions heard and appeared to be very candid in their feedback. It was clear that the face-to-face interviews provided the data necessary to demonstrate course outcomes.
The interview data suggest that a good time for follow-up would be three to four months after the course. This is a good time to learn about efforts already underway and/or to reinforce course principles by reminding graduates of the things they want to do.
This approach to demonstrating course impact is supported by the literature, is called for by the Booze Allen Hamilton Review of ELO courses, and intuitively makes a lot of sense. Senior leaders should be able to tell us whether or not they have applied what they learned and whether this investment benefits the Navy.
An issue that will need to be determined is how the interview data will be stored and analyzed in the future. This must be done in a way that findings and trends may be easily accessed by VADM Quast and his staff.
Recommendations
We recommend that the ELO adopt the recommendations that follow.
1. Revise the current Monterey NCBC evaluation forms as described above.
2. Implement the post-course impact evaluation protocol shown in Appendix C. Conduct face-to-face interviews 3-4 months after each NCBC on 5-6 graduates from across the spectrum of billets, in rotating fleet concentration areas.
3. Automate, manage, and analyze the data using a qualitative data analysis system such as “NVIVO.”
4. Create regular reports to demonstrate trends over time and to have— as needed at any given time—a database of findings on course impact. 5. Use this work as a prototype for evaluation in the other ELO classes.