Variables e instrumentos utilizados en la investigación

Apprenticeship, and WIA Youth Programs Will Likely Require a Different Set of Efficiency Measures

An often recurring message of ETA program administrators and state program operators – reinforced by the Expert Panel and findings from the literature – is that the 11 ETA programs have varying goals/objectives, target and serve different at-risk populations, offer widely varying types and intensities of services, and have widely differing costs. As a result, efficiency

measures need to be cautiously developed and tailored to what programs are attempting to achieve – and great care is needed in comparing results on such measures within programs (i.e., across states, grantees, and local jurisdictions) and across programs. The quantitative analyses of efficiency measure results conducted as part of this study, showed substantial differences on measures such as cost per entered employment both within programs (across states/grantees) and

between programs. Program officials at the state and local levels were concerned about unfair comparisons being made between programs and the possibility that higher cost programs (particularly those providing training services and serving more disadvantaged populations) would be seen in a poor light (e.g., have much higher cost per entered enrollment) – and perhaps, as a result, eventually see funding and services gravitate toward lower intensity, less costly services or have their budgets inappropriately reduced.

As alluded to in Recommendation #2, among the 11 programs included in this study, three programs in particular – the Apprenticeship, WIG, and the WIA Youth programs -- stand out as qualitatively different from the other programs. With regard to these three programs, ETA should be cautious in applying the measures identified in Recommendation #2 and should

consider alternative measures carefully tailored to the goals/operations of these programs:

• Apprenticeship Program – Several operational aspects of the Apprenticeship program make it quite different from the other 10 ETA programs. First, federal funding is used to support promotion, assessment, and registration of Apprenticeship programs across the country, but federal funding does not pay directly for delivery of services or training costs. The costs of training and serving apprentices are paid by the state, unions, employers, and/or participants. Second, in roughly half the states, these functions are undertaken by the states themselves at their own expense rather than ETA’s, and so in those states, there is virtually no federal cost. In addition, the Apprenticeship program encourages sponsors and apprentices to use WIA and other funding sources to cover the cost of training where appropriate – which means that costs of delivery of training is not fully reflected in Apprenticeship expenditures. Finally, because apprentices can be enrolled for up to five years, the costs in the year of completion may be less representative of the total costs when compared with other ETA programs.

• Work Incentive Grant Program – Unlike the other ETA programs, the WIG program does not directly enroll or serve individual customers. Rather, the Disability

Navigators (DNs) funded under this initiative are responsible for building the capacity of state and local workforce agencies to outreach to and more effectively serve individuals with disabilities. The primary role of DNs is to train staff at the local workforce level on disability issues and effective service delivery for disabled individuals. The grants issued under the WIG program are aimed at systemic change – Disability Navigators funded under the grants provide training to staff that is expected to yield improvements in access and quality of services to disabled

individuals over the length of the grant period (and even after the grant is concluded). There are several other factors that complicate the implementation of measures

recommended under Recommendation #2 – (1) funding for this program is scheduled to end shortly – and so it is likely that measures could not be fully implemented prior to the end of the program, (2) the current WIG reporting system relies on the

WIASRD data, which only covers WIA customers – which means that the effects of grants are missed for other (non-WIA enrolled) customers of the One-Stop system (e.g., Wagner-Peyser/ES registrants); and (3) the disability status of individuals served under WIA and other ETA programs is not fully captured because some individuals may not view themselves as disabled and/or are reluctant to reveal their disability status.

• WIA Youth Program - While employment, job retention, and earnings change are important goals for older youth served under the program, such goals are longer-term objectives for younger youth. Employment may, in fact, be counterproductive to younger participants compared to completing school and earning education credentials. Efficiency measures that capture employment and

education/credentialing are more appropriately connected to the underlying multiple purposes of the WIA Youth program component. An additional challenge with regard to the WIA Youth program is that expenditure data are only available at the national level for all youth combined (i.e., so it is not possible to calculate a separate efficiency measure for younger and older youth served by the program).

Reasonable alternatives for measuring efficiency are needed for the these three programs that take into consideration special circumstances with regard to program goals, types of

individuals served, and services delivered. While it is important to hold these three programs accountable for their “efficiency,” the measures employed need to be carefully linked to purposes and activities on which the federal funding is expended. Alternative efficiency measures for these three programs are the following:

• Recommendation #3a: ETA should consider alternative efficiency measures for the

Apprenticeship Program linked to the goals of the program and what federal funds are being spent on – for example, increasing the number of apprenticeships offered, building the quality of apprenticeship programs, and registering and monitoring of Apprenticeship programs accurately and in a timely manner. Therefore, ETA should consider applying the following alternative efficiency measures to the Apprenticeship program (all at the national level): cost per additional apprenticeship program registered and timeliness of registration decisions. Specific measures should be adopted after appropriate dialogue and analysis is undertaken.

• Recommendation #3b: ETA should consider alternative efficiency measures for the

WIG Program that reflect the training and technical assistance goals of this program. In particular, efficiency of this program should be aimed at measuring how the

services of Disability Navigators increase the numbers of disabled individuals served by the One-Stop system (and various ETA programs operating out of the One-Stop

system), as well as improvements in identifying individuals served by the workforce system and enhancements to the quality of services provided to disabled individuals. Therefore, ETA should consider applying the following alternative efficiency measure to the WIG program): cost per change in the number of One-Stop customers served with disabilities. As for the Apprenticeship program, efficiency measures for WIG should be adopted after appropriate dialogue and analysis. Because the WIG program may be terminated, it may be appropriate not to develop efficiency measures for this program.

• Recommendation #3c: As discussed in Recommendation #2 above, the efficiency

measure that should be applied to the WIA Youth Program is cost per placement in employment or education. In the longer term, the possibility of collecting separate cost and customer data for in-school and out-of-school youth should be investigated so that appropriate separate outcome and efficiency measures can be developed for these disparate groups. For example, if separate cost data were available for these two groups, it would feasible and perhaps appropriate to apply the four measures

recommended for the WIA Adult and Dislocated Worker Programs to the older youth served by the WIA Youth Program (i.e., cost per entered employment, cost per retained in employment, cost divided by post-program earnings, and cost divided by pre/post- program earnings).

There are distinctive characteristics of each program that make it difficult to

appropriately compare and contrast efficiency measure results across programs – and hence, ETA should be cautious in making such comparisons, and when direct comparisons are made, care should be taken to note that there are critical differences in program goals, types of individuals served, and types/intensity of services delivered under the various programs.

4. Recommendation #4: Performance Standards for States/Grantees on