ANEXO B INSTRUMENTO

The Expert Panel provided many useful comments at a meeting held November 9, 2009, and the panel also provided written comments, which are contained in Appendix D. The primary message conveyed by the panel is that based on the limited data available, difficult issues on how to deal with factors such as co-enrollment and shared costs, and ETA’s prior experience with efficiency measures (where an evaluation indicated that the efficiency measures led to creaming applicants and more use of less expensive services than was warranted), ETA should move very cautiously in adopting efficiency measures. Instead, the panel suggested that ETA track a variety of potential measures before establishing any efficiency measures associated with rewards or sanctions for states.

Dr. Beryl Radin’s comments stem from her perspective as a political scientist who has studied and written about performance measurement for many years. Radin’s central point is that the change in administrations in Washington has changed the views of OMB managers and

others on how performance management and program evaluation should be implemented. The changes in philosophy have not yet fully worked their way down to agency-specific measures, so she urges caution in continuing to apply the principles established by the prior administration. Moreover, implementation of the stimulus package has changed priorities about the goals of programs, leading to more focus on job creation than was previously the case. Finally, Radin suggests that ETA consider the basic objectives of the performance management system, and she wonders if it is prudent to use the same measures across programs with diverse services and customers.

Dr. Christopher King, a professor and economist who has been involved in performance management issues for workforce investment programs from their inception in the 1970s,, also urges ETA to be cautious in implementing efficiency measures for its programs. He notes that when efficiency measures were first employed, they led to serious problems and resulted in amendments prohibiting their use under JTPA. King also notes that the world is much more complex than it was when efficiency measures were first used because of co-enrollment,

leveraged resources, and loose measurement of performance data. He concludes that unintended consequences are all but guaranteed in such an environment. King urges that efficiency

measures with rewards and sanctions not be instituted at this time, and he argues that several programs, Apprenticeship and Work Incentive Grants, are so different that they should be

excluded from the measures. King makes several other points, and he suggests that while Return on Investment (ROI) be tracked, it should be looked at periodically through evaluations rather than through the performance management system on an annual basis.

Dr. Carolyn Heinrich, a public policy professor, provided detailed comments on all the issues raised for the meeting. While recognizing that it is important for ETA to explore the use

of outcome-based efficiency measures, Heinrich notes that the literature is clear that organizations respond to performance measures, often in unintended ways. She notes that conducting analyses at the state level hides much of the variation in outcomes and explanatory variables, so she recommends that data be collected and analyzed at the sub-state level. Like the other panel members, Heinrich stresses the importance of moving slowly before implementing efficiency standards. Heinrich emphasizes the importance of collecting uniform and accurate data on participants and also trying to collect efficiency data by activity.

Dr. Jeffrey Smith, an economics professor, advised ETA to be careful in interpreting efficiency measures of the type being considered. Smith notes that available research indicates that the outcome measures currently in use are measures of gross outcome only and are weakly, at best, related to program impact. Smith suggests that, in the long run, ETA should substantially modify the performance management system so that random assignment based impact estimates be computed for each state and that program impact be used in place of gross outcomes in performance measures. Smith is optimistic, based on his experience with Canadian workforce programs, that reasonable impact measures can be developed. Like the other panel members, Smith cautions that the WIG and Apprenticeship programs are quite different from the other nine programs and should have different efficiency measures. Smith provided detailed responses to the questions posed to the panel, and he urged ETA to develop accurate, consistent data as an important step in improving the performance management system.

In sum, the panel encourages ETA to move cautiously in adopting outcome-based efficiency standards. The panel encourages ETA to improve the quality of the data and to explore the used of sub-state data and activity-based data. In conjunction with these recommendations, the panel suggests that ETA track a number of measures to better assess

which measures provide the best efficiency measures. The panel members also offered

suggestions on how to deal with thorny issues such as co-enrollment, shared costs, and services to customers who are not counted in the outcome measures.

B. STUDY RECOMMENDATIONS ON IMPLEMENTATION OF OUTCOME-