Sugerencias:

MEASURES WIA (Excluding Youth), Wagner-Peyser, TAA, SCSEP, INA, WIG, and NFJP.

(Note: All of these programs use Common Measures.)

Entered Employment: Of those

who are not employed at the date of participation, the number of participants who are employed in the first quarter after the exit quarter divided by the number of

participants who exit during the quarter.

Employment Retention: Of

those who are employed in the first quarter after the exit quarter, the number of participants who are employed in both the second and third quarters after the exit quarter divided by the number of participants who exit during the quarter.

Average Earnings: Of those

participants who are employed in the first, second, and third quarters after the exit quarter, total earnings in the second quarter plus total earnings in the third quarter after the exit quarter divided by the number of participants who exit during the quarter. WIA Youth (Note: The Youth program is covered by Common Measures, but uses difference measures.) Placement in Employment or Education: Of those who are not in

postsecondary education or

employment (including the military) at the date of participation, the number of youth participants who are in employment (including the military) or enrolled in

postsecondary education and/or advanced training/occupational skills training in the first quarter after the exit quarter divided by the number of youth participants who exit during the quarter.

None None Attainment of a Degree or

Certificate: Of those enrolled in

education (at the date of

participation or at any point during the program), the number of youth participants who attain a diploma, GED, or certificate by the end of the third quarter after the exit quarter divided by the number of youth participants who exit during the quarter.

Literacy and Numeracy: Gains

of those out-of-school youth who are basic skills deficient, the number of youth participants who increase one or more educational functioning levels divided by the number of youth participants who have completed a year in the program (i.e., one year from the date of first youth program service) plus the number of youth

participants who exit before completing a year in the program.

EXHIBIT 6-1: DEFINITIONS AND USE OF OUTCOME MEASURES ACROSS ETA PROGRAMS

PROGRAM EMPLOYMENT MEASURES EMPLOYMENT

RETENTION MEASURES EARNINGS MEASURES

EDUCATION/SKILLS ATTAINMENT OR OTHER MEASURES Apprenticeship Program (Apprenticeship is transitioning from these measures to the Common Measures)

None Employment Retention: The

number of apprentices employed nine months after registration divided by the number of apprentices registered in the first quarter of the fiscal year.

Earnings Gain: The difference

between the average of the current wage of the total number of entrants still employed nine months later and the average of the starting wage of the total number of entrants registered in the first quarter of the fiscal year.

Average Cost Per Registered Apprentice: Program budget

allocation divided by total active federal program participants (apprentices).

Note: The aim of Common Measures is to have common performance measures for programs with similar goals. Originally, ETA and VETS collaborated to define Common Measures, and as of July 1, 2005, these measures were implemented for the WIA Title IB and Wagner-

Peyser/Employment Services and VETS programs; TAA programs implemented the Common Measures on October 1, 2005. INA, SCSEP, and NFJP transitioned to the Common Measures reporting system in PY 2006. Common Measure outcomes are calculated for the WIG program using WIASRD data -- for each measure, the result is calculated for WIA enrolled individuals with disabilities for the workforce areas receiving

Disability Program Navigator (DPN) cooperative agreements. The Apprenticeship is making the transition to the Common Measures framework (from the performance measures shown in the table).

Source: U.S. Department of Labor website for the Glossary of Performance. More detailed discussion of each of these measures is available also in Training and Employment Guidance Letter No. 17-05, “Common Measures Policy for the Employment and Training Administration’s (ETA) Performance Accountability System and Related Performance Issues,” issued by DOL/ETA, February 17, 2006,and a change to this original guidance.

attain additional educational degrees/certifications and improvements in numeracy and literacy proficiency.

The WIG program is also covered by the Common Measures displayed in Exhibit 6-1, but because the WIG program has no enrolled customers, the measures must be implemented somewhat differently.53 For each measure, the rate is calculated for WIA enrolled exiters with disabilities served in workforce areas receiving Disability Program Navigator (DPN) grants. This means that other customers of the One-Stop system not enrolled in WIA, such as Wagner- Peyser exiters (i.e., not co-enrolled in WIA), go uncounted under the Common Measures for the WIG program.54 When interpreting performance and efficiency measures for the WIG program, it is important to take into consideration that the program is primarily a capacity building

program aimed at improving access and service delivery for individuals with disabilities within the workforce development system. Unlike the other Common Measure programs, individuals are not formally enrolled in the WIG program or served directly by Disability Navigators.55

Apprenticeship is in the process of transitioning from the measures shown in Exhibit 6-1 to the Common Measures. During FY 2009, the Office of Apprenticeship (OA) is collecting baseline data to support the transition to the Common Measures. Three quarters of data collected through Common Reporting Information System (CRIS) will be reviewed and analyzed. Targets for entered employment, retention, and average earnings will be set using the baseline results for

53_{The program has set annual performance goals for each of the three Common Measures, as well as for a fourth} measure – percentage of exiters with disabilities in the workforce areas receiving a DPN cooperative agreement. 54_{In a February 11, 2008 Memorandum to Round One and Two DPN states, notes: “Although the Wagner-Peyser} program registers many more individuals than those identified in the WIASRD, ETA is unable to derive data at the local workforce area level to identify impacts of Navigators located there.”

55_{To further complicate matters, disability status is not always disclosed by individuals served under WIA, ES,} TAA, and other programs – and there is considerably variability across programs in the percentage of individuals who in fact disclose whether they have a disability. Individuals with disabilities are often co-enrolled in a number of ETA programs and a range of other programs (particularly Vocational Rehabilitation) – so accounting for

participation and tying participation to costs is complicated. Finally, with regard to cost, the budget for WIG is a fraction of the budget of programs such as WIA and ES, and costs of the WIG program (particularly related to supporting the more than 500 Disability Navigators across the country) is supplemented by funds from the Social Security Administration and states (e.g., in New York, an estimated half the costs of Disability Navigators is paid by the state). Finally, funding for the WIG program has not been renewed and the program will end in the coming year.

apprentices who complete training programs during subsequent reporting years.56 As shown in Exhibit 6-1, the performance measures used in the past by Apprenticeship, while including measures of employment and earnings, are somewhat different from those mandated under the Common Measures. For example, the Apprenticeship Program has not in the past used a

measure of entered employment. The Apprenticeship program has used the concept of retention in employment, but the definition and way in which “retention” has been calculated are different from that of the Common Measures, i.e., retention in the Apprenticeship program has been defined as “the number of apprentices employed nine months after registration divided by the number of apprentices registered in the first quarter of the fiscal year.”

As also shown in the exhibit, while the Apprenticeship program has used an earnings measure, unlike the Common Measures, the earnings measure has been one of “earnings gain” and is substantially different in nature from the Common Measures’ post-program “average earnings” measure, i.e., earnings gain in the Apprenticeship program has in the past been defined as “the difference between the average of the current wage of the total number of entrants still employed nine months later and the average of the starting wage of the total number of entrants registered in the first quarter of the fiscal year.”57 There are also three other issues (discussed earlier in this report) regarding Apprenticeship that make the program different from the other ETA programs and should be taken into consideration in establishing efficiency measures appropriate to this program: (1) generally, costs of training and serving apprentices are paid by the state, unions, employers, and/or participants (i.e., so federal costs of the program are low and generally are targeted on promoting Apprenticeship and helping to ensure that programs offered are of high quality); (2) apprentices are enrolled for up to five years, so costs in the year of exit

56_{Specifically, OA received the first set of data from the CRIS; two additional quarters of data are expected} December 2008 and January 2009; and a draft of the targets is expected to be available by June 2009.

may be much less representative of the total cost than in other programs; and (3) some states administer their apprenticeship program themselves, so there are no federal costs.

Overall, the current measures of participation and outcome collected by the 11 ETA programs suggest that at least in the short run, a feasible and potentially cost-effective strategy for implementing efficiency measures would be adopting efficiency measures that build on the structure of the outcome Common Measures being collected and reported by ETA programs. As will be discussed later in this report in the study’s recommendations (see Chapter 8),

considerable caution is required and there are a number of serious methodological and data constraints that will need to be resolved before valid and reliable efficiency measurement can be implemented by the 11 ETA programs.

B. ANALYSES OF EFFICIENCY MEASURES RESULTS BASED ON THE COMMON MEASURE OUTCOMES AT THE NATIONAL LEVEL

ETA program offices provided data on Common Measure performance outcomes, and the Office of Financial and Administrative Management (OFAM) provided data on program

expenditures to support development of estimates of efficiency measure results for PY/FY 2007. The estimates provided in this section and the next section (on state-level results) are preliminary and (as discussed throughout this report) great care should be taken in making cross program comparisons. For several program it was also possible to collect cost and outcome data that permitted three years of analyses of efficiency measure results.58 Exhibit 6-2 provides

preliminary analysis of results for efficiency measure results for candidate efficiency measures (based on the Common Measures) at the national level for PY/FY 2007.

58 _{OFAM provided three years (PY 2005 through PY 2007) of expenditure data for the 11 ETA programs; Common} Measure outcome results for three years were provided for the WIA (Adults, Dislocated Worker, Youth, and NEG), Wagner-Peyser, and TAA programs. NFJP was able to provide PY 2007 outcome data, but not for earlier because the program had only started to track Common Measure outcomes in PY 2007. Outcome data was not available for INAP, WIG, or the Apprenticeship program.

EXHIBIT 6-2: ESTIMATED RESULTS FOR FOUR OUTCOME-BASED EFFICIENCY