TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 7-99

TO:         ALL STATE WORKFORCE LIAISONS
            ALL STATE WORKER ADJUSTMENT LIAISONS
            ALL STATE EMPLOYMENT SECURITY LIAISONS
            ALL ONE STOP CAREER-CENTER SYSTEM LEADS
FROM:	    Lenita Jacobs-Simmons
            Deputy Assistant Secretary  
SUBJECT:    Core and Customer Satisfaction Performance Measures
            for the Workforce Investment System

1. Purpose. To provide technical guidance for States to implement the core and customer satisfaction performance measures and calculate performance levels required under the Workforce Investment Act (WIA).

2. References. Workforce Investment Act of 1998 (Pub. L. 105-220), Section 136, Workforce Investment Act, Interim Final Rule, 20 CFR Part 666, published at 64 Federal Register 18662 (April 15, 1999).

3. Background. Section 136 of the WIA specifies core indicators of performance for workforce investment activities in adult, dislocated worker, and youth programs. Fifteen core measures apply to the adult, dislocated worker and youth programs, and two measures of customer satisfaction apply across these three funding streams for a total of 17 required measures. The measures specified in the Act are as follows:

Adult Program

1. Entry into unsubsidized employment;
2. Retention in unsubsidized employment six months after entry into the employment;
3. Earnings received in unsubsidized employment six months after entry into the employment; and
4. Attainment of a recognized credential relating to achievement of educational skills, which may include attainment of a secondary school diploma or its recognized equivalent, or occupational skills, by participants who enter unsubsidized employment.

Dislocated Worker Program

5. Entry into unsubsidized employment;
6. Retention in unsubsidized employment six months after entry into the employment;
7. Earnings received in unsubsidized employment six months after entry into the employment relative to earnings of job of dislocation; and
8. Attainment of a recognized credential relating to achievement of educational skills, which may include attainment of a secondary school diploma or its recognized equivalent, or occupational skills, by participants who enter unsubsidized employment.

Youth Program

Older Youth (aged 19-21)

9. Entry into unsubsidized employment;
10. Retention in unsubsidized employment six months after entry into the employment;
11. Earnings received in unsubsidized employment six months after entry into the employment; and
12. Attainment of a recognized credential relating to achievement of educational skills, which may include attainment of a secondary school diploma or its recognized equivalent, or occupational skills, by participants who enter unsubsidized employment or who enter postsecondary education, advanced training or unsubsidized employment.

Younger Youth (aged 14-18)

13. Attainment of basic skills and, as appropriate, work readiness or occupational skills;
14. Attainment of secondary school diplomas and their recognized equivalents; and
15. Placement and retention in postsecondary education, advanced training, military service, employment, or qualified apprenticeships.

Across Funding Streams

16. Customer satisfaction for participants; and
17. Customer satisfaction for employers.

The core performance measures are the key measures of success in achieving the legislative goals of WIA. The measures are used to: set agreed upon performance goals on a State and local level; ensure comparability of state performance results to maintain objectivity in measuring results for incentive and sanction determinations; and provide information for system wide reporting and evaluation for program improvement. States and localities may also need other measures for ongoing oversight, program management, and continuous improvement purposes. It should be noted that some measures will be used by other related programs. For example, the Trade Adjustment Assistance and NAFTA - Transitional Assistance programs (mandatory partners of WIA) will use dislocated worker measures discussed herein.

Section 136 of the Act also calls for the use of quarterly Unemployment Insurance (UI) wage records to measure performance. There is a significant time delay in availability of the UI wage record for use as a data source. To address the time-lags associated with the UI wage records, States will be required to go back in time, prior to WIA implementation, to both simulate WIA performance levels for developing baseline data for negotiations and calculating performance measures for WIA implementation. For most of the core measures, the performance levels reported will also lag behind. This means that outcomes for most participants will not be reported during the same year that individuals participate in services.

This guidance includes the following sections:

� a discussion of the performance and accountability framework which identifies key concepts that apply across all of the measures.
� the core measures that apply to the adult, dislocated worker, and youth funding streams. This includes a discussion of operational definitions of the measures and data sources and methods to collect data on outcomes.
� the customer satisfaction measures that apply across funding streams to participants and employers. It presents a detailed description of the methodology for implementing the customer satisfaction measures.
� the attachments, which include: A) WIA Core and Customer Satisfaction Measures-at-a-Glance, B) Definitions of Key Terms, and C)Guidance for Calculating WIA Core Measures.

4. Performance and Accountability Framework.

A. Applying the WIA Core Measures Across Partner Programs. A major emphasis of the legislation is improving coordination between the workforce investment system and adult education, literacy, and vocational rehabilitation programs that were re-authorized in the bill and vocational education programs addressed in separate legislation. Incentive grants will be awarded based on exceeding State adjusted levels of performance for WIA Titles I and II (Adult Education and Literacy) and the Carl D. Perkins Vocational and Applied Technology Education Act. The Department is working with the U.S. Department of Education to develop a coordinated performance accountability framework across labor, adult education, vocational education and vocational rehabilitation programs. Initially, these efforts are focusing on the common components of the core measures. These efforts will also include developing common terms and additional measures which States and localities may choose to draw upon.

To encourage integration of services and recognize shared contributions toward outcomes, the Department will use the following strategy for tracking and reporting across WIA Title I funded (WIA funded programs) and other workforce investment partners. Once an individual has registered for WIA funded services, programs can claim credit for outcomes on core measures attained by participants who receive non-WIA funded services (such as those offered by One-Stop and School-to-Work partner programs). The non-WIA-funded partner programs that are included in the performance measurement system must be party to local Memorandums of Understanding (MOU). Each partner program must have the capacity to track registered WIA participants until the individual exits (the definition of exit is in Section 4.D) all WIA and non-WIA-funded partner services. Some methods for tracking participants across programs include specifying the non-WIA funded services in the individuals' service plans, coordinating services across WIA and non-WIA funded services, and providing follow-up services to individuals.

B. Applying Measures to Funding Streams and Population Groups. The WIA core measures, other than the customer satisfaction measures, are applied separately to the three funding streams: adult, dislocated workers, and youth. The youth funding stream is further divided into two categories: older youth (aged 19-21) and younger youth (aged 14-18). This means that 15 of the measures are divided among the four different populations groups. The two customer satisfaction measures are applied across all population groups. Since the outcomes vary across the four population groups, the Department maintains that it is important to measure performance separately for each group. The following rules apply to determine how to count participants in the funding streams:

• If a participant is served by a specific funding stream, he/she will be counted in that funding stream's set of measures (e.g., a participant served by adult funds will be in the adult measures).

• If a participant is served by more than one funding stream, he/she will be counted in more than one set of measures (e.g., a participant served by youth funds and adult funds will be counted in the youth measures and adult measures).

• Since there are two sets of measures within the youth funding stream, the following rule applies for those served by the youth funding stream: a youth must be included only in the set of youth measures that applies based on the person's age at registration (i.e., youth between the ages of 14 and 18 at registration will be included in the younger youth measures and youth between the ages of 19-21 at registration will be included in the older youth measures) regardless of how old the participant is at exit (this is a different rule for counting youth in the measures than the rule for counting youth used in PY 1999 by the early implementing States).

• All exit-based measures (which includes all the core measures except the younger youth skill attainment rate and the employer customer satisfaction measure) will be effective when the participant has exited all WIA Title I and partner services (definition of exit is provided in Section D). In the case of a youth participant who becomes an adult participant, the individual will be counted in both the youth measures and adult measures. Neither set of exit-based measures will be applied until that participant has exited the adult program services (as well as all other WIA Title I or partner services). This is because exit-based measures assess a participant's outcome following their full range of services, even when those services are from different program areas.

C. Point of Registration for Counting Customers in Measures. For accountability purposes, WIA established core measures of performance for customers who receive workforce investment activities beyond self-service and informational services. The WIA Interim Final Rule (Section 666.140) stated that the point of registration determines who is counted in the measures and who is excluded from the measures. All youth who receive WIA Title I services will be registered for services and counted in the measures.

For performance measurement of the adult and dislocated worker programs, WIA distinguishes self-service and informational activities as separate from the other activities within the WIA service categories (core, intensive, and training). However, there are informational activities within the core services category described in the Act. There are two main factors to consider when determining which core services require adults and dislocated workers to be registered and counted in the measures:

1. Level of staff involvement with the customer. When there is significant staff involvement in terms of resources or time, individuals receiving the staff-intensive core services are required to be registered for the adult and dislocated worker programs (all youth customers are required to register).
2. Purpose of the service. The Act specifically excludes those individuals who participate in self-service activities only (such as browsing the Internet). For staff-assisted activities, the purpose of the service should be examined to determine if registration is required for the service. Services that are designed to inform and educate individuals about the labor market and their employment strengths, weaknesses, and the range of services appropriate to their situations should be considered informational in nature. Staff-assisted services that are designed to impart job seeking and/or occupational skills should require registration.

For example, individuals receiving the following categories of core services should be registered.

• Staff assisted job search and placement assistance, including career counseling
• Staff assisted job referrals (such as testing and background checks)
• Staff assisted job development (working with employer and job-seeker)
• Staff assisted workshops and job clubs

Using these criteria, Table 1 presents the WIA core, intensive, and training services that require registration on a Federal level and the core informational/self-service activities that do not require registration. This table includes the required WIA services specified in the Act for all of the categories and identifies some finer distinctions for service categories such as job search and placement assistance. Required services specified in the Act are represented in Table 1 in italics.

It is important to keep in mind that an individual must receive a WIA funded staff-assisted core, intensive, or training service to trigger registration and include the person in the core measures. For example, in a One-Stop environment, if an individual received only Wagner Peyser funded core services or TANF funded case management, and did not receive any WIA funded staff-assisted core, intensive, or training service, that individual would not be registered for WIA and counted in the core measures. However, as long an individual has received WIA funded staff-assisted core, intensive, or training and goes on to receive non-WIA funded partner services, that person would be registered for WIA and counted in the core measures.

D. Point of Exit for Counting Outcomes. For all of the core measures (except the younger youth skill attainment rate and employer customer satisfaction measure), comparability across States is only possible if a single point in time is used to begin measurement. The term "exit" is being used to determine when to count an individual in a specified reporting period. Each individual becomes part of an exit cohort, a group who are determined to be "exiters" within a particular quarter and are looked at together for measurement purposes. There are two ways to determine exit during a quarter:

1. a participant who has a date of case closure, completion or known exit from WIA-funded or non-WIA funded partner service within the quarter (hard exit) or

2. a participant who does not receive any WIA-funded or non-WIA funded partner service for 90 days and is not scheduled for future services except follow-up services (soft exit).

Participants who have a planned gap in service of greater than 90 days should not be considered as exited if the gap in service is due to a delay before the beginning of training or a health/medical condition that prevents an individual from participating in services. Service providers should document any gap in service that occurs with a reason for such a gap in service. Participants who exit from services because they are incarcerated, deceased or have a health/medical condition that prevents the individual from participating in services, should be excluded from the measures. Once a participant has not received any WIA funded or partner services for 90 days, except follow-up services and there is no planned gap in service or the planned gap in service is for reasons other than those specified above, that participant has exited WIA for the purposes of measurement in 15 of the 17 core measures (the younger youth skill attainment rate and employer customer satisfaction measures are not based on exit).

The exit date is the last date of WIA funded or partner service received (except follow-up services). For a soft exit, the exit date cannot be determined until 90 days has elapsed from the last date of service. At that point, the exit date recorded is the last date of service. The exit quarter (referred to throughout the definitions of the measures) is the quarter in which the last date of service (except follow-up services) takes place. If a participant exits WIA and receives additional WIA services after exiting, that participant is treated as a new participant for purposes of the core measures and will be included in the appropriate measures. The definition of exit applies to all of the core measures except the younger youth skill attainment and employer customer satisfaction measures.

E. Time Period to be Used for Measurement. As far as time periods for measurement are concerned, there are two groups of measures. The first group of measures is related to employment as the primary outcome or as a partial outcome and require the use of the UI wage records as a data source. These measures are: the adult, dislocated worker and older youth entered employment, retention, earnings, and credential measures and the younger youth retention rate. The second group of measures are not employment based and are not tied to the UI wage records. These measures are: the younger youth skill attainment rate and diploma or equivalency rate, and the two customer satisfaction measures.

A major consideration for the first group of employment-related measures is the time lag associated with using the UI wage records as a data source. For the most part, there is a time delay of at least two quarters after the quarter in which a person exits services and the time when all the UI wage record data for the quarter after exit are available for record matching. For measures that occur at six months after exit, such as the employment retention and earnings measures, the delay will be at least one year after the exit quarter. A detailed chart of the timing of activities and the measurement time frame is included in Attachment C, Exhibit II.1.

To address the time-lags in the use of the UI wage records, States will need to go back in time, prior to WIA implementation to calculate performance measures for actual PY 2000 WIA implementation. This will only need to be done for the group of employment-related measures that use the UI wage records. Since there is no need to go back in time for the second group of measures, these non-employment related measures will use the actual program year time period, beginning on July 1, 2000. The two time periods for measurement in PY 2000 are as follows:

1. The first time period is for all measures that utilize the UI wage record and will involve going back in time three quarters to account for the time-lag. These measures include the adult, dislocated worker, and older youth entered employment, retention, earnings, and credential measures and the younger youth retention rate. For PY 2000, all States will use the last three quarters of PY '99 and the first quarter of PY 2000 for all the measures listed above.

2. The second time period is for the four measures that are not based on the UI wage records: the younger youth skill attainment rate, the younger youth diploma or equivalency rate, and the two customer satisfaction measures. For PY 2000, these four measures will be "real time" measures in which States will use the four quarters in PY 2000 for purposes of calculating performance.

This approach of having two time periods, with one going back three quarters prior and the first quarter of the current program year and the second for the group of "real-time" measures being for the actual program year, will continue for future years of WIA implementation.

F. Participants to be Excluded from the Measures. Participants who exit from the program under the following conditions should be excluded from the measures:

• For JTPA participant data, exclude those who received objective assessment only and participants in Title II 8 % funded programs.
• Participants who exit from services because they are incarcerated, institutionalized, deceased or have a health/medical condition that prevent them from participating in services, should be excluded from the measures.
• For JTPA participant data, exclude dislocated workers served exclusively with National Reserve Account (NRA) funds. For WIA participant data, exclude dislocated workers served exclusively with National Emergency Grants.

Additional exclusions may apply to certain measures. These exclusions are identified in the specific definitions of applicable measures in Section 5.H.

G. Operational Definitions of the Core Performance Measures. All of the definitions of the measures are written as an equation, clearly identifying who is in the numerator and who is in the denominator. In cases where there are conditions that apply to both the numerator and denominator, the condition is represented in italics at the beginning of the measure. For example: The adult entered employment rate is defined as:

Of those who are not employed at registration: the number of adults who have entered employment by the end of the first quarter after exit divided by the number of adults who exit during the quarter.

The condition "of those not employed at registration" applies to both the numerator and denominator as follows: the number of adults who are not employed at registration and enter employment by the end of the first quarter after exit divided by the number of adults who are not employed at registration and exit during the quarter. In addition, the phrase "who exit during the quarter" is used frequently in the denominators of the measures. This phrase refers to the exit quarter.

Operational parameters and definitions of key terms are also provided to further clarify details needed to implement the measures. Since many of the measures are identical or similar across funding categories, every attempt has been made to define the measures as consistently as possible. This means that identical or similar measures use the same time periods and consistent operational parameters to the extent possible. A two page summary of the measures in a calculation format is provided in Attachment A. Definitions of all the key terms identified in this document are included in Attachment B. Detailed guidelines for calculating the measures are provided in Attachment C.

5. Core Performance Measures that Apply to the Adult, Dislocated Worker, and Youth Funding Streams

H. Adult Measures

Measure 1: Adult Entered Employment Rate

Of those who are not employed at registration:

Number of adults who have entered employment by the end of the first quarter after exit divided by the number of adults who exit during the quarter.

Operational Parameters:

• Individuals who are employed at registration are excluded from this measure (i.e. programs will not be held accountable for these individuals under this measure).
• Employment at registration is based on information collected from the registrant, not on UI wage records.

Rationale:

While the Department recognizes concerns that have been raised about excluding incumbent and underemployed workers from this measure, the intent of the legislation clearly focuses on entering employment. It becomes very difficult to include individuals who are employed at registration without further complicating the measure and making it subject to misinterpretation. Individuals who are employed at registration can demonstrate positive outcomes in the retention, earnings gain/replacement and combination employment and credential measure.

Definitions:

Employed at Registration - An individual employed at registration is one who, during the 7 consecutive days prior to registration, did any work at all as a paid employee, in his or her own business, profession or farm, worked 15 hours or more as an unpaid worker in an enterprise operated by a member of the family, or is one who was not working, but has a job or business from which he or she was temporarily absent because of illness, bad weather, vacation, labor-management dispute, or personal reasons, whether or not paid by the employer for time-off, and whether or not seeking another job.

Not Employed at Registration - An individual who does not meet the definition of employed at registration.

Employed in Quarter After Exit Quarter - The individual is considered employed if UI wage records for the quarter after exit show earnings greater than zero. UI Wage records will be the primary data source for tracking employment in the quarter after exit. When supplemental data sources are used, individuals should be counted as employed if, in the calendar quarter after exit, they did any work at all as paid employees (i.e., received at least some earnings), worked in their own business, profession, or worked on their own farm.

Measure 2: Adult Employment Retention Rate at Six Months

Of those who are employed in the first quarter after exit:

Number of adults who are employed in the third quarter after exit divided by the number of adults who exit during the quarter

Operational Parameters:

• This measure includes only those who are employed in the first quarter following exit (regardless of their employment status at registration).
• Individuals who are not found to be employed in the first quarter after exit are excluded from this measure (i.e., programs will not be held accountable for these individuals under this measure).
• Employment in the first and third quarters following exit does not have to be with the same employer.

Rationale:

While the proposed measure does not necessarily indicate continuous employment for six months, it does meet the intent of the Act by showing that the person is employed six months after entering employment. This measure requires verifying employment in the quarter after exit rather than relying on employment status at registration to be in aligned with the U.S. Department of Education, Vocational Rehabilitation services measure of employment retention.

Measure 3: Adult Average Earnings Change in Six Months

Of those who are employed in the first quarter after exit:

Total post-program earnings (earnings in quarter 2 + quarter 3 after exit) minus pre-program earnings (earnings in quarter 2 + quarter 3 prior to registration) divided by the number of adults who exit during the quarter.

Operational Parameters:

• This measure includes the same population as the adult employment retention measure, those who are employed in the first quarter following exit (regardless of their employment status at registration).
• To ensure comparability of this measure on a national level, the UI wage records will be the only data source for these measure. Therefore, individuals whose employment in either the first or third quarter after exit was determined from supplementary sources and not from the UI wage records are excluded from the measure (i.e., programs will not be held accountable for these individuals under this measure).
• Individuals who are not found to be employed in the first quarter after exit are excluded from this measure.
• States should exclude from this measure any individuals whose entry (registration) date is so far back in time that accessing quarters 2 and 3 of pre-registration wage data is unfeasible or unreasonable. However, participants excluded from this measure for this reason should still be included in any other applicable measures. For example, the person should still be counted in the retention measure.
• If supplementary sources are the data source for a participant's employment in the 2^nd and/or 3^rd quarter prior to registration, that participant is excluded from this measure.

Rationale:

This measure is designed as a pre- and post-program look at earnings change. Since the legislation specifies earnings at six months after employment, a six month period was selected for the pre-program comparison. The 2^nd and 3^rd quarters prior to registration were selected as the pre-program reference period because trends have shown that many program participants experience intermittent or stop-gap employment immediately prior to participating in employment and training programs.

Measure 4: Adult Employment and Credential Rate

Of adults who received training services:

Number of adults who were employed in the first quarter after exit and received a credential by the end of the third quarter after exit divided by the number of adults who exited services during the quarter.

Operational Parameters:

� The numerator of this measure includes those who were employed in the first quarter after exit regardless of whether they were employed at registration.
� Credentials can be obtained while a person is still participating in services and up to three quarters following exit.

Definitions:

Training Services - include WIA-funded and non-WIA funded partner training services. These services include: occupational skills training, including training for nontraditional employment; on-the-job training; programs that combine workplace training with related instruction, which may include cooperative education programs; training programs operated by the private sector; skill upgrading and retraining; entrepreneurial training; job readiness training; adult education and literacy activities in combination with other training; and customized training conducted with a commitment by an employer or group of employers to employ an individual upon successful completion of the training.

Credential - nationally recognized degree or certificate or State/locally recognized credential. Credentials include, but are not limited to, a high school diploma, GED or other recognized equivalents, post-secondary degrees/certificates, recognized skill standards, and licensure or industry-recognized certificates. States should include all State Education Agency recognized credentials. In addition, States should work with local Workforce Investment Boards to encourage certificates to recognize successful completion of the training services listed above that are designed to equip individuals to enter or re-enter employment, retain employment, or advance into better employment.

Rationale:

This measure is limited to individuals who are in training because that is the only set of services that lead to attainment of a credential. To promote program integration and partner collaboration, this measure does recognize joint participation in WIA services and non-WIA funded training programs. While there will be variation in the ability of local providers or state agencies to track participants who are enrolled in non-WIA funded training services, it is considered a significant and necessary step toward building an integrated workforce funded programs will be able to get credit for non-WIA funded training in addition to WIA funded training.

This measure includes all individuals who received training regardless of whether they had jobs at registration. It recognizes not only individuals who enter employment, but also incumbent workers, students and welfare-to-work participants, who have jobs or are placed in jobs, but continue receiving training services to attain more skills and better jobs. A time frame of three quarters after exit to obtain a credential is proposed to allow time for individuals to take tests, which may occur after training is completed and may only be offered once a year.

I. Dislocated Worker Measures⁽²⁾

Measure 5: Dislocated Worker Entered Employment Rate

Number of dislocated workers who have entered employment by the end of the first quarter after exit divided by the number of dislocated workers who exit during the quarter

Operational Parameters:

� All dislocated workers are included because dislocated workers are either not employed or scheduled to lose their primary job for which eligibility for the dislocated worker program was based.

Rationale:

This measure uses the same measurement quarters as the adult entered employment rate. The only distinction is that all dislocated workers are counted in this measure regardless of their employment status at registration.

Measure 6: Dislocated Worker Employment Retention Rate at Six Months

Of those who are employed in the first quarter after exit:Number of dislocated workers who are employed in the third quarter after exit divided by the number of dislocated workers who exit during the quarter.

Operational Parameters:

� Employment in the first and third quarters following exit does not have to be with the same employer.
� Individuals who are not found to be employed in the first quarter after exit are excluded from this measure (i.e., programs will not be held accountable for these individuals under this measure).

Rationale:

This measure has the same rationale as the adult employment retention rate.

Measure 7: Dislocated Worker Earnings Replacement Rate in Six Months

Of those who are employed in the first quarter after exit:

Total post-program earnings (earnings in quarter 2 + quarter 3 after exit) divided by the pre-dislocation earnings (earnings in quarters 2 + quarter 3 prior to dislocation)

Operational Parameters:

� To ensure comparability of this measure on a national level, the UI wage records will be the only data source for this measure. Individuals whose employment in either the first or third quarter after exit was determined from supplementary sources and not from the UI wage records are excluded from the measure (i.e., programs will not be held accountable for these individuals under this measure).
� This measure includes the same population as the retention measure, those who are employed in the first quarter following exit.
� Individuals who are not found to be employed in the first quarter after exit are excluded from this measure.
� If there is no date of dislocation or if the date of dislocation is after registration, use the 2^nd and 3^rd quarters prior to registration. If a State is not systematically collecting the date of dislocation and uses the 2^nd and 3^rd quarters prior to registration date, it should be noted that the registration date may not closely simulate the results from the 2^nd and 3^rd quarters prior to the dislocation quarter.
� States should exclude from the earnings replacement calculations any individuals whose entry (registration) date is so far back in time that accessing quarters 2 and 3 pre-dislocation/pre-registration wage data is unfeasible or unreasonable. However, participants excluded from this measure for this reason should still be included in any other applicable measures. For example, these participants should still be counted in the retention measure.
� The calculation for this indicator will be done on an aggregate basis. The Department will continue to research the feasibility of calculating this measure on an individual basis using wage record data.

Definitions:

Date of dislocation - The last day of employment at the dislocation job. If there is no date of dislocation, date of registration will be used instead.

Rationale:

As opposed to the adult earnings measure that calculates post-program earnings increases as compared to pre-program earnings, the earnings replacement rate, which computes the percentage of pre-program earnings being earned post-program, is being used for dislocated workers. This is because it may be difficult to find dislocated workers (with formerly high paying jobs) a new job with equivalent or better wages; therefore measuring earnings increases would result in negative numbers and would be not be representative of a dislocated worker's true success. Because a major goal of the dislocated worker program is to ameliorate earnings loss as well as to replace or increase earnings, a better measure of the dislocated worker program's effectiveness is percentage of earnings of the new job in relation to the job of dislocation. It was further determined that the quarter prior to dislocation often includes severance pay or may reflect reduced hours. This is why quarters 2 and 3 prior to dislocation were chosen.

Measure 8: Dislocated Worker Employment and Credential Rate

Of dislocated workers who received training services:

Number of dislocated workers who were employed in the first quarter after exit and received a credential by the end of the third quarter after exit divided by the number of dislocated workers who exited services during the quarter

Operational Parameters:

• The numerator of this measure includes those who were employed in the first quarter after exit regardless of whether they were employed at registration.
• Credentials can be obtained while a person is still participating in services.
• See adult employment and credential measure definitions for training services and credentials.

Rationale:

The rationale for this measure is the same as for the adult employment and credential measure.

J. Older Youth (Age 19-21) Measures

Measure 9: Older Youth Entered Employment Rate

Of those who are not employed at registration and who are not enrolled in post-secondary education or advanced training in the first quarter after exit:

Number of older youth who have entered employment by the end of the first quarter after exit divided by the number of older youth who exit during the quarter

Operational Parameters:

� Individuals who are employed at registration are excluded from this measure (i.e., programs will not be held accountable for these individuals under this measure).
� Employment at registration is based on information collected from the registrant, not on UI wage records.
� Individuals in both employment and post-secondary education or advanced training in the first quarter after exit will be included in the denominator. Individuals who are not employed, but are in only post-secondary education or advanced training in the first quarter after exit are excluded from this measure (i.e., programs will not be held accountable for these individuals under this measure).

Definitions:

Post-Secondary Education - a program at an accredited degree-granting institution that leads to an academic degree (e.g., AA, AS, BA, BS). Does not include programs offered by degree-granting institutions that do not lead to an academic degree.

Advanced Training - an occupational skills employment/training program, not funded under Title I of the WIA, which does not duplicate training received under Title I. Includes only training outside of the One-Stop, WIA and partner, system (i.e., training following exit).

Rationale:

The exclusion of those individuals who move on to post-secondary education or advanced training and not employment avoids the disincentive of penalizing a program for placing an older youth in post-secondary education or advanced training. It better aligns the older youth measures with the younger youth measures which reward programs for moving youth into post-secondary education or advanced training. Those individuals who are excluded from this measure due to entry into post-secondary education or advanced training will be measured in the older youth credential rate.

Measure 10: Older Youth Employment Retention Rate at Six Months

Of those who are employed in the first quarter after exit and who are not enrolled in post-secondary education or advanced training in the third quarter after exit:

Number of older youth who are employed in third quarter after exit divided by the number of older youth who exit during the quarter.

Operational Parameters:

� This measure includes all individuals who are employed in the first quarter following exit, except those individuals who are employed in the first quarter and not employed in the third quarter following exit, but are in post-secondary education or advanced training third quarter following exit. These individuals are excluded from this measure (i.e., programs will not be held accountable for these individuals under this measure).
� Employment in the first and third quarters following exit does not have to be with the same employer.

Rationale:

As in the entered employment rate, it would be unfair to penalize a participant who was employed in the quarter after exit, but left employment to start post-secondary education in the third quarter after exit. Therefore, those not employed in the third quarter after exit who are in post-secondary education or advanced training in that third quarter following exit are excluded from the measure.

Measure 11: Older Youth Average Earnings Change in Six Months

Of those who are employed in the first quarter after exit and who are not enrolled in post-secondary education or advanced training in the third quarter after exit:

Total post-program earnings (earnings in quarter 2 + quarter 3 after exit) minus pre-program earnings (earnings in quarter 2 + quarter 3 prior to registration) divided by the number of older youth who exit during the quarter.

Operational Parameters:

� This measure includes the same population as the older youth employment retention measure (regardless of their employment status at registration).
� To ensure comparability of this measure on a national level, the UI wage records will be the only data source for this measure. Therefore, individuals whose employment in either the first or third quarter after exit was determined from supplementary sources and not from UI wage records are excluded from this measure (i.e., programs will not be held accountable for these individuals under this measure).
� Individuals who are not found to be employed in the first quarter after exit are excluded from this measure.
� States should exclude from this measure any individuals whose entry (registration) date is so far back in time that accessing quarters 2 and 3 of pre-registration wage data is unfeasible or unreasonable. However, participants excluded from this measure for this reason should still be included in any other applicable measures. For example, the person should still be counted in the retention measure.
� If supplementary sources are the data source for a participant's employment in the 2^nd and/or 3^rd quarter prior to registration, that participant is excluded from this measure.

Rationale:

See rationale for adult earnings change.

Measure 12: Older Youth Credential Rate

Number of older youth who are in employment, post-secondary education, or advanced training in the first quarter after exit and received a credential by the end of the third quarter after exit divided by the number of older youth who exit during the quarter

Operational Parameters:

� As opposed to the adult and dislocated worker measures where a credential must be coupled with employment, for older youth, a credential can be coupled with employment, entry into post-secondary education, or entry into advanced training.
� As opposed to the adult and dislocated worker measures where only those who received training services are included in the measure, all older youth exiters will be included in this measure.
� Credentials can be obtained while a person is still participating in services.

This denominator is different from the adult and dislocated worker credential measure in that those two measures are a subset of participants enrolled in training, while this measure includes all older youth. The reason is that there is not a specific activity of "training" in the youth program as there is in the adult and dislocated worker programs through individual training accounts. Therefore, it is not possible to limit the measure to those enrolled in training.

K. Younger Youth (Age 14-18) Measures

Measure 13: Younger Youth Skill Attainment Rate

Of all in-school youth and any out-of-school youth assessed to be in need of basic skills, work readiness skills, and/or occupational skills:

Total number of basic skills goals attained by younger youth plus number of work readiness skills goals attained by younger youth plus number of occupational skills goals attained by younger youth divided by the total number of basic skills goals plus the number of work readiness skills plus the number of occupational skills goals set.

Operational Parameters:

� The measure should create an appropriate intermediate-type measure for youth who require more services, such as academic and soft skills development, prior to attaining a diploma or equivalency, employment, and post secondary education.
� If a participant is deficient in basic literacy skills, the individual must set, at a minimum, one basic skills goal (the participant may also set work readiness and/or occupational skills goals, if appropriate).
� WIA participants counted in this measure will be all in-school; and any out-of-school youth assessed to be in need of basic skills, work readiness skills, and/or occupational skills.
� All youth measured in this rate must have a minimum of one skill goal set per year and may have a maximum of three goals per year.
� The target date for accomplishing each skill goal must be set for no later than one year.
� The skill goal or the target date set can only be extended if the participant has a gap in service where they are placed in a hold status in which the participant is not receiving services but plans to return to the program. When they enter a hold status, the one year clock for the goal target date stops. The clock begins again once the participant is no longer in a hold status.
� Goals will fall into the category of basic skills, work readiness skills, or occupational skills. Participants may have any combination of the three types of skill goals (three skill goals in the same category, two skill goals in one category and one skill goal in another, or one skill goal in each category, etc.).
� Success of skill attainment goals will be recorded in the quarter of goal achievement, while failure will be recorded in the quarter one year from the time the goal was set if not attained by such time.

Definitions:

Basic literacy skills deficient - the individual computes or solves problems, reads, writes, or speaks English at or below the 8^th grade level or is unable to compute or solve problems, read, write, or speak English at a level necessary to function on the job, in the individual's family, or in society. In addition, States and locals have the option of establishing their own definition, which must include the above language. In cases where States and/or locals establish such a definition, that definition will be used for basic literacy skills determination.

Basic skills goal - measurable increase in basic education skills including reading comprehension, math computation, writing, speaking, listening, problem solving, reasoning, and the capacity to use these skills.

Occupational skills goal - primary occupational skills encompass the proficiency to perform actual tasks and technical functions required by certain occupational fields at entry, intermediate or advanced levels. Secondary occupational skills entail familiarity with and use of set-up procedures, safety measures, work-related terminology, record keeping and paperwork formats, tools, equipment and materials, and breakdown and clean-up routines.

Work readiness skills goal - work readiness skills include world of work awareness, labor market knowledge, occupational information, values clarification and personal understanding, career planning and decision making, and job search techniques (resumes, interviews, applications, and follow-up letters). They also encompass survival/daily living skills such as using the phone, telling time, shopping, renting an apartment, opening a bank account, and using public transportation. They also include positive work habits, attitudes, and behaviors such as punctuality, regular attendance, presenting a neat appearance, getting along and working well with others, exhibiting good conduct, following instructions and completing tasks, accepting constructive criticism from supervisors and co-workers, showing initiative and reliability, and assuming the responsibilities involved in maintaining a job. This category also entails developing motivation and adaptability, obtaining effective coping and problem-solving skills, and acquiring an improved self image.

Rationale:

There is concern about the potential lack of comparability among States and local areas on this measure. At the same time, local flexibility is important because of the individual service strategy philosophy. With these competing forces, it is important to set some guidelines to ensure comparability, while not being so proscriptive as to prevent participants from setting appropriate goals. Therefore, a maximum of three goals per person in each year is allowable in order to prevent the setting of multiple minimum-level skills goals, while still encouraging participants to set goals in any one of the skill categories.

Measure 14. Younger Youth Diploma or Equivalent Attainment

Of those who register without a diploma or equivalent:

Number of younger youth who attained secondary school diploma or equivalent by the end of the first quarter after exit divided by the number of younger youth who exit during the quarter (except those still in secondary school at exit)

Operational Parameters:

� If a younger youth exits WIA while still enrolled in secondary education, the individual is excluded from the measure (i.e., programs will not be held accountable for these individuals under this measure).
� All younger youth (except those still in secondary school at exit and those who have already attained their diploma or equivalent prior to registration) will be assessed in this measure in the quarter after exit.

Definitions:

High School Diploma Equivalent - a GED or high school equivalency diploma recognized by the State.

Rationale:

Those participants who exit WIA while still enrolled in secondary education are excluded from the measure. This exclusion is because some participants may decide to leave the program before finishing high school. It would be impossible for those participants to have attained a diploma or equivalent at the time of exit if they are still enrolled in high school. This measure is intended to motivate program staff to continue services to participants until they attain a diploma or its equivalent.

Measure 15. Younger Youth Retention Rate

Number of younger youth found in one of the following categories in the third quarter following exit:

-- post secondary education
-- advanced training
-- employment
-- military service
-- qualified apprenticeships

divided by the number of younger youth who exit during the quarter (except those still in secondary school at exit).

Operational Parameters:

� If the participant is in one of the placement activities listed above during the third quarter following exit, the individual is counted as successfully retained (the participant does not have to remain in the same activity for the entire retention period, as long as the participant is found in one of the activities during the third quarter).
� If the participant exits WIA and does not enter into any of placement activities by the time retention is measured, the participant is counted in the denominator of the measure and it is reflected as a negative outcome.
� If a younger youth exits WIA while still enrolled in secondary education, the individual is excluded from the measure (i.e., programs will not be held accountable for these individuals under this measure).

Definitions:

Qualified apprenticeship - a program approved and recorded by the ETA/Bureau of Apprenticeship and Training (BAT) or by a recognized State Apprenticeship Agency (State Apprenticeship Council). Approval is by certified registration or other appropriate written credential.

Military service - reporting for active duty.

Rationale:

This measure assesses retention in the third quarter following exit from the program. This measurement time period is consistent with the adult and dislocated worker retention period, which also measure retention in the third quarter following exit. The WIA stipulated one measure for younger youth placement and retention. Because of the increasing emphasis on post-program measures, retention was chosen as the focus of this measure. To be consistent with the diploma/equivalency attainment rate and because it would be unfair to penalize those participants who exit while still in secondary school, the denominator of this measure does not include those who exit while still in secondary school.

L. How Summer Youth Activities fit into Performance

The Measures. Unlike the performance measurement system under the Job Training Partnership Act (JTPA) where participants in the separately funded summer youth program were not included in measuring JTPA youth outcomes, all WIA youth will be measured as part of a comprehensive youth program assessed by the core performance measures for youth. The majority of youth who have participated in the summer program in the past have been between the ages of 14-17. These participants would fall under the three younger youth measures; the skill attainment rate, the diploma/equivalency measure and the retention rate.

An in-school youth who exits and returns to secondary school following participation in summer employment opportunities would not be included in the younger youth diploma or equivalency rate or the younger youth retention rate. Such a participant would only be included in the skill attainment rate. Within the skill attainment rate, if the participant is basic literacy skills deficient, the individual must have a basic skills goal. If the participant is not basic literacy skills deficient, a work readiness skills goal would be appropriate for a youth in a summer employment opportunity. Therefore, it is in the best interest of the local board to serve an in-school youth who is basic skills deficient all year long and not only in the summer employment opportunities component.

Out-of-school youth would be included in all three younger youth measures because they would not be returning to secondary school following summer employment opportunities. Therefore, it is in the best interest of participants and of the local board to serve out-of-school youth all year long and not only in summer employment opportunities.

All older youth are included in the four older youth measures: entered employment, retention, earnings change and credential attainment, with exceptions in certain circumstances. All four of these measures are outcome measures assessed following exit. Older youth who participate in the summer portion of the youth program and exit will be held to the same rules as any other older youth who exits the program. Therefore, it is in the interest of local areas to serve older youth who participate in summer employment opportunities in year-round services if the youth do not have a credential and/or job placement.

Youth who transition from JTPA to WIA during the summer. All JTPA youth who transition into WIA on July 1 will be included in the WIA youth performance measures. The following section applies to those in-school youth who were enrolled in JTPA and transition into WIA. Service providers may assess youth enrolled in JTPA Title II-B in the spring of 2000 as if they were WIA participants, in anticipation of them becoming WIA participants on July 1, 2000. This means determining basic skills needs and setting skill attainment goals for these youth. Alternatively, youth competencies established under JTPA may be converted into skill attainment goals so that they will count toward the skill attainment rate. Reassessment should only take place if the youth never received a basic skill level determination.

Examples of how to convert youth competencies into skill attainment goals include the following: basic education skills under youth competencies would be the equivalent of a basic skills goal under the skill attainment measure, pre-employment skills and work maturity skills under youth competencies would be the equivalent of a work readiness skills goal under the skill attainment measure, and job-specific skills under youth competencies would be the equivalent of an occupational skills goal under the skill attainment measure.

M. Data Sources and Methods to Collect Data on Adult, Dislocated Worker, and Youth Measures. This section describes data sources and methods to collect data for all of the measures except the two customer satisfaction measures (see Section 6. for description of customer satisfaction methodology and data sources). Since there are common outcomes across most of the adult, dislocated worker and youth measures, the data sources are described by outcome (i.e.,, employment, credential, etc.). Table 2 is presented to help easily reference the measures by the outcome category and data sources.

1. Employment

Unemployment Insurance Wage Records

To the extent it is consistent with State law, the Unemployment Insurance (UI) wage records will be the primary data source for tracking entered employment, employment retention, earnings change/replacement, the employment portion of the credential rate for adults, dislocated workers, and older youth and the employment portion of the younger youth retention rate. In addition to UI wage records within a State, the Wage Record Interchange System (in the pilot stages at this time) and other State Employment Security Offices (to track UI wage records across States) are considered acceptable wage record sources. If individuals are not found in the UI wage record sources, States may use supplemental data sources for the entered employment, retention, credential and younger youth retention measures. However, the only data source that can be used for the earnings change/replacement measures is the UI wage records. Further guidance on the use of UI wage records is provided in Attachment C.

Supplemental Data Sources

While the majority of employment in a State's workforce is "covered" and will be in the UI wage records, certain types of employers and employees are excluded by Federal unemployment law standards or are not covered under a State's UI law. "Uncovered" employment typically includes Federal employment, postal service, military, railroad, out-of-state employment, self employment, some agricultural employment, and some employment where earnings are primarily based on commission. States have flexibility in methods used to obtain information on participants in "uncovered" employment. Examples include:

1) Case management, follow-up services, and surveys of participant to determine that the participant is employed and written documentation of that employment; or

2) Record sharing and/or automated record matching with other employment and administrative databases to determine and document employment. These databases may include:

�Office of Personnel Management (Federal Career Service);
�United States Postal Service;
�Railroad Retirement System;
�U.S. Department of Defense;
�State Department of Revenue or Tax (State income tax for self-reported occupations); and
�Government Employment Records (State government, local government, judicial employment, public school employment, etc.);

Most States will likely utilize case management, follow-up services, and surveys of participants with written documentation for supplemental data. Some States already have record sharing and/or automated matching systems in place that they will be able to use to track "uncovered" employment. For States that do not have these systems in place, developing these relationships with State and Local agencies is encouraged. However, record sharing of Federal databases can best be coordinated on a national level rather than having each individual State contact these agencies. It is also possible for States to establish record sharing/matching agreements with agencies such as the State Corrections Department to determine whether individuals are incarcerated and should be excluded from the measures. Even though record sharing is an efficient and reliable data source, many of these agencies may only provide data annually so that it would not be available for quarterly reporting.

All data and methods to supplement wage record data must be documented and are subject to audit. Computer records from automated record matching are considered a valid written record. A telephone response from the participant must be accompanied by a written document such as W2 form, pay stub, 1099 form, or other written documentation. Telephone verification of employment with employers is acceptable, but must also be documented. For self-employed individuals, telephone verification with major clients/contracting entities is also acceptable, but must be documented. Supplemental data must be recorded within 30 days after the individual was found missing in the wage record.

2. Credential Attainment

States and localities will have flexibility in methods used to collect data on credentials for the adult, dislocated worker, older youth credential rate, and younger youth diploma/equivalent attainment rate. Examples include:

1) Case management, follow-up services, and surveys of participant to determine that the participant has received a credential and written documentation of that credential; or

2) Record sharing agreements and/or automated record matching with administrative/other databases to determine and document that the participant has received a credential. These databases may include:

� State Board of Education;
� State Board Governing Community Colleges;
� State Board Governing Universities;
� State Licensing Boards for Private Schools;
� State Education Associations, Integrated Postesecondary Education Reporting Unit;
� Higher Education Planning Unit;
� State Department of Professional or Occupational Regulation (possibly other units such as health care administration or specific boards like the "Board of Nursing");
� Employers;
� Training Institutions/Providers; and
� Adult Basic Education Providers (GED/Equivalent testing agencies)

Most States will likely utilize case management, follow-up services, and surveys of participants with written documentation of credentials. Some States already have record sharing and/or automated matching systems in place that they will be able to use to track credentials.

All data and methods to determine achievement of credentials must be documented and are subject to audit. Computer records from automated record matching are considered a valid written record. A telephone response from the participant must be accompanied by written document such as a certificate, degree, or other written documentation. Telephone verification with the certification institution/entity that a person has attained the credential is also acceptable, but must also be documented.

3. Placement in Post-Secondary Education or Advanced Training

The older youth credential rate and the younger youth retention rate have multiple potential outcomes. In both cases, one of the potential outcomes is placement in post-secondary education or advanced training. The younger youth retention rate also has other possible outcomes. Therefore, the data sources and methods for data collection are described later in this section under "Younger Youth Retention." In addition to the older youth credential rate, the other three older youth employment-related measures also allow for individuals to be excluded from those measures if the youth is found in post-secondary education or advanced training outside of the WIA funded and partner system. The following data sources can be used to determine whether youth are placed in post-secondary education or advanced training.

1) Case management, follow-up services, and surveys of the participant to determine if the youth has been placed in post-secondary education or advanced training and written documentation of that placement; or

2) Record sharing agreements and/or automated record matching with administrative/other databases to determine and document that the participant has been placed in post-secondary education or advanced training. These databases may include:

� State Board Governing Community Colleges;
� State Board Governing Universities;
� State Education Associations;
� Integrated Postsecondary Education Reporting Unit;
� Higher Education Planning Unit; and
� Training Institutions/Providers

Most States will likely utilize case management, follow-up services, and surveys of participants with written documentation. Some States already have record sharing and/or automated matching systems in place that they will be able to use to track outcomes. All data and methods must be documented and are subject to audit. Computer records from automated record matching are considered a valid written record. A telephone response from the participant must be accompanied by written documentation. Telephone verification with the appropriate institution/entity is also valid, but must be documented.

4. Skill Attainment

To measure skill attainment for the younger youth skill attainment measure, local programs must assess youth at intake to determine whether they are in need of basic skills, work readiness skills, and/or occupational skills. This assessment will include a review of basic skills, occupational skills, prior work experience, employability, interests, aptitudes and supporting service needs. From this assessment, skill attainment goals will be established (see operational parameters under Section I, skill attainment measure). To determine whether youth meet the skill attainment goals will require a pre-assessment and post-assessment of skill level. The use of a standardized assessment procedure such as a standardized test or a performance-based assessment with a standardized scoring method is encouraged. These tests may include:

� Test of Adult Basic Education (TABE);
� DOL Workplace Literacy Test (DOL-WLT);
� Adult Measure of Educational Skills (AMES);
� Adult Basic Learning Examination (ABLE);
� Adult Literacy Test (ALT), Simon & Schuster;
� Armed Forces Qualifying Test (AFQT);
� Basic Occupational Literacy Test (BOLT);
� California Achievement Test (CAT);
� Career Ability Placement Survey (CAPS);
� Comprehensive Adult Student Assessment System (CASAS);
� General Aptitude Test Battery (GATB);
� Iowa Test of Basic Skills (ITBS);
� Metropolitan Achievement Test (MAT);
� Reading Job Corps Screening Test (RJCST); or
� Wide Range Achievement Test (WRAT)

In cases where a standardized test or a performance-based assessment is not available (such as for work-readiness skills), assessment techniques must be objective, unbiased and conform to widely accepted, clearly defined criteria, be field tested for utility, consistency, and accuracy, and provide for the training/preparation of all raters/scorers. Information on achievement of skill attainment goals should be derived from case management or follow-up services. All data and methods to determine achievement of skill attainment goals must be documented and are subject to audit.

5. Younger Youth Retention

Since the younger youth retention rate has five possible positive outcomes, multiple data sources are needed to track participants. The data sources to determine whether a youth has been placed in employment or military service are addressed in the previous discussion on data sources for employment. The data sources to determine whether a youth has been placed in post-secondary education or advanced training are addressed in the previous discussion on placement in post-secondary education or advanced training. To determine whether a youth has been placed in a qualified apprenticeship, the following methods can be used:

1) Case management, follow-up services, and surveys of the participant and written documentation; or

2) Record sharing agreements and/or automated record matching with the U.S. Department of Labor, Bureau of Apprenticeship and Training or recognized State Apprenticeship Agency .

Most States will likely utilize case management, follow-up services, and surveys of participants with written documentation. Some States already have record sharing and/or automated matching systems in place that they will be able to use to track outcomes. All data and methods to determine achievement of one of the five outcomes must be documented and are subject to audit. Computer records from automated record matching are considered a valid written record. A telephone response from the participant must be accompanied by written documentation. Telephone verification with the appropriate institution/entity is valid, but must also be documented.

6. Customer Satisfaction Measures.

A. Overview of Measurement Approach. To meet the customer satisfaction measurement requirements of WIA, the Department will use customer satisfaction surveys. The survey approach that will be utilized allows State and local flexibility and, at the same time, captures common customer satisfaction information that can be aggregated and compared at a State and national level. This will be done through the use of a small set of required questions that will form a customer satisfaction index. The Department will use the American Customer Satisfaction Index (ACSI), which is created by combining scores from three specific questions that address different dimensions of customers' experience. For WIA application, there will be one score for each of the two customer groups: participants and employers.

The ACSI is the most widely used index currently in practice. It is used extensively in the business community, including over 150 Fortune 500 companies, and in many European countries. Twenty-nine agencies of the Federal government are using the ACSI. In addition, it has been used twice in the past four years to assess customer satisfaction for ETA's Quality Initiative, the Enterprise. The ACSI will allow the workforce investment system to not only look at performance within the system, but also be able to gain perspective on the workforce system's performance by benchmarking against organizations and industries outside of the system. The ACSI also has a history of being useful in tracking change in customer satisfaction over time, making it an ideal way to gauge States' progress "toward continuously improving in performance."

Since the ACSI trademark is proprietary property of the University of Michigan and its software is owned by Claes Fornell International (CFI) Group, the Department has established a license agreement with the University of Michigan that will allow States the use of the ACSI for a Statewide sample of participants and employers. States that want to use the ACSI for measuring customer satisfaction for each local area will have to establish an independent contract with the University of Michigan. States may also contract with CFI Group for additional assistance in measuring, analyzing, and understanding ACSI data.

B. Customer Satisfaction Measures

Measure 16: Participant Satisfaction

The weighted average of participant ratings on each of the three questions regarding overall satisfaction are reported on a 0-100 point scale. The score is a weighted average, not a percentage.

1. Who Will Be Surveyed?

WIA Title I-B participants who are exiters as defined in the core measures, who are either Adults, Dislocated Workers, Youth 19-21, or Youth 14-18 will be surveyed. All individuals from all funding streams in an exit cohort are eligible to be chosen for inclusion in the random sample.

2. How Many (number obtained)?

Except in small States, a sample will be taken from these exiter groups in each quarter. Five hundred completed participant surveys must be obtained each year for calculation of the indicator. A completed participant survey is defined as a survey in which all three questions regarding overall satisfaction have been answered. For small States (those with less than 1000 exiters in a year), the entire population must be surveyed. The response rate from the sample with valid contact information must be a minimum of 50 percent. The standard of 500 from a sample of the whole population of customers provides accuracy such that there is only a 5 in 100 chance that the results would vary by more than �5 points from the score obtained from surveying the whole population.

3. How (methodology)?

The responses are obtained using a uniform telephone methodology. The rationale for only using telephone surveys include: the comparability of the indicator for assessing performance levels is most reliably obtained with a telephone survey; telephone surveys are easily and reliably administered; and defining procedures for mailed surveys is more difficult than defining procedures for telephone surveys. Estimates of the cost of telephone surveys nationwide run an average of $15 per completed survey. Since it is being proposed that States complete 500 participant and 500 employer surveys, the cost would be an estimated total of about $15,000 per State per year.

4. When to Conduct Surveys?

The surveys should be conducted on a rolling basis within the time frame for participants and employers indicated below. To obtain sufficient numbers, smaller States will need to survey on an ongoing basis. Participants should be contacted within 60 days of the exit date or the date that an exit date has been determined. This means either 60 days after the date of an exit interview or 60 days after the 90 days have elapsed since the last service date.

5. What are the Questions?

A survey conducted by telephone in which the following questions are asked at the beginning of the interview⁽³⁾:

My name is ____________ with XXXXX and I am conducting a survey for the XXXX XXXXX. I would like to speak to Ms./Mr.___________.

Are you the Ms./Mr. _________ who was looking for a job a few months ago?

I would like to ask you some questions about your recent experience looking for a job. Our purpose is to learn from you how to improve programs and services offered to people in XXX. The survey should take about XX minutes to complete. First I am going to read a list of services you may have received. Indicate as I read them those you recall receiving during the period in which you were seeking employment and/or training at the XX center.

� A thorough assessment of your needs
� Assistance about finding a job
� Assistance to develop an individual employment plan
� Assistance to decide about the best training to take
� Assistance from someone to support you during your job search or training
� Did you receive any training?
� Occupational training
� Training to give you general skills for the workplace
� Training in English or math

1). Utilizing a scale of 1 to 10 where "1" means "Very Dissatisfied" and "10" means "Very Satisfied" what is your overall satisfaction with the services provided from ________?

Very Dissatisfied                         Very Satisfied     DK(4)     REF(5)
1     2     3     4     5     6     7     8     9     10     11        12

2). Considering all of the expectations you may have had about the services, to what extent have the services met your expectations? "1" now means "Falls Short of Your Expectations" and "10" means "Exceeds Your Expectations."

Falls Short of                                   Exceeds     DK     REF
1     2     3     4     5     6     7     8     9     10     11      12

3). Now think of the ideal program for people in your circumstances. How well do you think the services you received compare with the ideal set of services? "1" now means "Not very close to the Ideal" and "10" means "Very Close to the Ideal."

Not Close to Ideal                   Very Close to Ideal     DK     REF
1     2     3     4     5     6     7     8     9     10     11      12

Measure 17: Employer Satisfaction

The weighted average of employer ratings on each of the three questions regarding overall satisfaction are reported on a 0-100 point scale. The score is a weighted average, not a percentage.

1. Who Will Be Surveyed?

Those eligible for surveying include employers who have received service where the service has been completed or, if it is an ongoing service, when a full segment of service has been provided (e.g., after listing an open job order, the employer has received some referrals or if no service, 30 days have elapsed after the initial request). All employers who have received a substantial service involving personal contact with One-Stop staff are eligible to be chosen for inclusion in the random sample (this excludes those employers who request a brochure or standard mailing, those who ask a question that is answered with little expenditure of staff time, or those who use electronic self-services).⁽⁶⁾ Examples of services include staff facilitated job orders, customized job training, customized labor market information requests, and on-the-job training activities.

When an employer has received multiple services, priority should be given to the service that required the greatest expenditure of funds or staff time⁽⁷⁾ and the survey conducted regarding their satisfaction with that service.

2. How Many (number obtained)?

Except in small States, a sample will be taken from these employers in each quarter. Five hundred completed participant surveys must be obtained each year for calculation of the indicator. A completed employer survey is defined as a survey in which all three questions regarding overall satisfaction have been answered. For small States (those with less than 1000 employers who received a substantial service in a year) the entire population must be surveyed. The response rate from the sample with valid contact information must be a minimum of 50 percent. The standard of 500 from a sample of the whole population of customers provides accuracy such that there is only a 5 in 100 chance that the results would vary by more than �5 points from the score obtained from surveying the whole population.

3. How (methodology)?

The responses are obtained using a uniform telephone methodology. The rationale for only using telephone surveys include: the comparability of the indicator for assessing performance levels is most reliably obtained with a telephone survey; telephone surveys are easily and reliably administered; and defining procedures for mailed surveys is more difficult than defining procedures for telephone surveys. Estimates of the cost of telephone surveys nationwide run an average of $15 per completed survey. Since it is being proposed that States complete 500 participant and 500 employer surveys, the cost would be an estimated total of about $15,000 per State per year.

4. When to Conduct Surveys?

The surveys should be conducted on a rolling basis within the time frame indicated. To obtain sufficient numbers, smaller States will need to survey on an ongoing basis. Employers should be contacted within 60 days of the completion of the service or 30-60 days after a job order has been listed where no referrals have been made.

5. What are the Questions?

A survey will be conducted by telephone in which the following questions are asked at the beginning of the telephone interview (the first question is a sample and can be modified to suit the needs of individual states):

A survey conducted by telephone in which the following questions are asked at the beginning of the interview⁽⁸⁾:

My name is ____________ with XXXXX and I am conducting a survey for the XXXX XXXXX. I would like to speak to Ms./Mr.___________.

Are you the Ms./Mr. _________ who (describe the service received).

I would like to ask you some questions about your recent experience with ________. Our purpose is to learn from you how to improve programs and services offered to employers. The survey should take about XX minutes to complete.

1). Utilizing a scale of 1 to 10 where "1" means "Very Dissatisfied" and "10" means "Very Satisfied" what is your overall satisfaction with the service(s) provided from ________?

Very Dissatisfied                         Very Satisfied     DK     REF
1     2     3     4     5     6     7     8     9     10     11      12

2). Considering all of the expectations you may have had about the services, to what extent have the services met your expectations? "1" now means "Falls Short of Your Expectations" and "10" means "Exceeds Your Expectations."

Falls Short of                                   Exceeds     DK     REF
1     2     3     4     5     6     7     8     9     10     11      12

3). Now think of the ideal service(s) for people in your circumstances. How well do you think the service(s) you received compare with the ideal service(s)? "1" now means "Not Very Close to Ideal" and "10" now means "Very Close to the Ideal."

Not Close to Ideal                   Very Close to Ideal     DK     REF
1     2     3     4     5     6     7     8     9     10     11      12

C. Definition of Terms

Sample. A group of cases selected from a population by a random process where everyone has an equal probability of being selected.

Response rate. The percentage of people who have valid contact information who are contacted and respond to the questions on the survey.

Valid contact information. Information that leads the interviewer or surveyor to the location where the contact individual is located whether or not the individual responds and answers the questions.

D. The Calculation

The calculation for the American Customer Satisfaction Indicator (ACSI) is accomplished by calculating the weighted average of the raw scores for each of the customer satisfaction questions given by each respondent. The weighted average score is then transformed to an index reported on a 0-100 scale. The aggregate index score is simply the weighted average of each case's index score.

Notes: CFI Group will provide the actual weights given for (W1), (W2), and (W3) below. In calculating respondent level index scores, round to two decimal points.

When calculating the average index score, round to the nearest whole number. For any case, the general formula for calculating the index score is given as:

Index Score = {[(Q1)(W1) + (Q2)(W2) + (Q3)(W3)] -1} x 11.111

where:
Q1 = raw score on question #1
Q2 = raw score on question #2
Q3 = raw score for question #3
W1 = weight for question #1
W2 = weight for question #2
W3 = weight for question #3

Example:

If the respondent answers were 5, 8, and 9 respectively for each of the three customer satisfaction questions, and the weights for each of the three questions were .4, .2, and .4* respectively the calculation for the respondent's index score would be as follows:

{[(5)(.4) + (8)(.2) + (9)(.4)] - 1} x 11.111 =

{[7.2] - 1} x 11.111 = 68.89

If two more respondents whose on the three questions were 6, 10, and 6 and 9, 6, and 7 respectively, using the same weights listed above, those two respondent's index scores would be:

64.44 and 73.33. To calculate the aggregate index score, simply average the individual respondent's index scores and round to the nearest whole number as follows:

68.89 + 64.44 + 73.33 /3 = 69

* These weights are examples only, CFI group will provide the actual weights.

7. Action Required. States shall distribute this Guidance Letter to all officials within the State who need such information to implement WIA core and customer satisfaction measures.

8. Inquiries. Questions concerning this issuance may be directed to your appropriate Regional Office.

9. Attachments.
A) WIA Core and Customer Satisfaction Measures-at-a-Glance
B) Definitions of Key Terms
C) Guidance for Calculating WIA Core Measures

RESCISSIONS

EXPIRATION DATE    Continuing

DISTRIBUTION

Attachment A - WIA Core Measures-at-a-Glance

Adult Measures

1. Entered Employment Rate

Of those who are not employed at registration:
# of adults who have entered employment by the end of the 1^st quarter (Qtr.) after exit
# of adults who exit during the quarter

2. Employment Retention Rate

Of those who are employed in 1^st Qtr. after exit:
# of adults who are employed in 3^rd Qtr. after exit
# of adults who exit during the quarter

3. Earnings Change in Six Months

Of those who are employed in 1^st Qtr. after exit:
[Total Post-Program Earnings (earnings in Qtr 2 + Qtr 3 after exit)] - [Pre-Program Earnings (earnings in Qtrs 2 + 3 prior to registration)]
# of adults who exit during the quarter

4. Employment and Credential Rate

Of those who received training services:
# of adults who were employed in the 1^st Qtr. after exit and received a credential by the end of 3rd quarter after exit
# of adults who exited services during the quarter

Dislocated Worker Measures

5. Entered Employment Rate
# of dislocated workers (DW) who have entered employment by the end if the 1^st Qtr. after exit
# of dislocated workers who exit during the quarter

6. Employment Retention Rate

Of those who are employed in the 1^st Qtr. after exit:
# of DW who are employed in 3^rd Qtr. after exit
# of DW who exit during the quarter

7. Earnings Replacement Rate
Of those who are employed in the 1^st Qtr. after exit:
Total Post-Program Earnings (earnings in Qtr 2 + Qtr 3 after exit)
Pre-Dislocation Earnings (earnings in Qtrs 2 +3 prior to dislocation)

8. Employment and Credential Rate

Of those who received training services:
# of DW who were employed in the 1^st quarter after exit and received a credential by the end of 3rd quarter after exit
# of DW who exit during the Qtr.

Older Youth (19-21 years old) Measures

9. Entered Employment Rate

Of those who are not employed at registration and who are not enrolled in post-secondary education or advanced training in the 1^st Qtr. after exit:
# of older youth (OY) who have entered employment by the end of the 1^st Qtr. after exit
# of OY who exit during the quarter

10. Employment Retention Rate

Of those who are not employed at registration and who are not enrolled in post-secondary education or advanced training in the 3^rd Qtr. after exit:
# of OY who are employed in 3^rd Qtr. after exit
# of OY who exit during the quarter

11. Earnings Change in Six Months

Of those who are employed in 1^st Qtr. after exit who are not enrolled in post-secondary education or advanced training in the 3^rd Qtr. after exit:
[Total Post-Program Earnings (earnings in Qtr 2 + Qtr 3 after exit)] - [Pre-Program Earnings (earnings in Qtrs 2 + 3 prior to registration)]
# of OY who exit during the quarter

12. Credential Rate
# of OY who were in employment, post-secondary education, or advanced training in the first Qtr. after exit and received a credential by the end of 3rd Qtr. after exit
# of OY who exit during the quarter

Younger Youth (14-18 years old) Measures

13. Skill Attainment Rate

Of all in-school youth and any out-of-school youth assessed to be in need of basic skills, work readiness skills, and/or occupational skills:

Total # of attained basic skills + # of attained WR skills + # of attained Occ. skills
Total # of basic skills goals + # of WR skills goals + # of Occ. skills goals

14. Diploma or Equivalent Attainment Rate

Of those who register without a diploma or equivalent:
# of younger youth (YY) who attained a secondary school diploma or equivalent by the end of the 1^st Qtr. after exit
# of YY who exit during the Qtr. (except those still in secondary school at exit)

15. Retention Rate

# of younger youth found in one of the following in the 3^rd Qtr. after exit:
-- post secondary education
-- advanced training
-- employment
-- military service
-- qualified apprenticeships
# of YY who exited during the Qtr. (except those still in secondary school at exit)

Across Funding Streams

16. Participant Customer Satisfaction

The weighted average of participant ratings on each of the 3 questions regarding overall satisfaction reported on a 0-100 scale.

17. Employer Customer Satisfaction

The weighted average of employer ratings on each of the 3 questions regarding overall satisfaction reported on a 0-100 scale.

Attachment B - Definitions of Key Terms

Advanced Training - an occupational skills employment/training program, not funded under Title I of the WIA, which does not duplicate training received under Title I. Includes only training outside of the one-stop, WIA and partner, system (i.e., training following exit).

Basic literacy skills deficient - the individual computes or solves problems, reads, writes, or speaks English at or below the 8^th grade level or is unable to compute or solve problems, read, write, or speak English at a level necessary to function on the job, in the individual's family, or in society. In addition, States and locals have the option of establishing their own definition, which must include the above language. In cases where States and/or locals establish such a definition, that definition will be used for basic literacy skills determination.

Basic Skills Goal - measurable increase in basic education skills include reading comprehension, math computation, writing, speaking, listening, problem solving, reasoning, and the capacity to use these skills.

Credential - nationally recognized degree or certificate or State/locally recognized credential. Credentials include, but are not limited to a high school diploma, GED or other recognized equivalents, post-secondary degrees/certificates, recognized skill standards, and licensure or industry-recognized certificates. States should include all State Education Agency recognized credentials. In addition, States should work with local Workforce Investment Boards to encourage certificates to recognize successful completion of the training services listed above that are designed to equip individuals to enter or re-enter employment, retain employment, or advance into better employment.

Date of Dislocation - The last day of employment at the dislocation job. If there is no date of dislocation, date of registration will be used instead.

Employed at Registration - An employed individual is one who, during the 7 consecutive days prior to registration, did any work at all as a paid employee, in his or her own business, profession or farm, worked 15 hours or more as an unpaid worker in an enterprise operated by a member of the family, or is one who was not working, but has a job or business from which he or she was temporarily absent because of illness, bad weather, vacation, labor-management dispute, or personal reasons, whether or not paid by the employer for time-off, and whether or not seeking another job.

Employed in Quarter After Exit Quarter - The individual is considered employed if UI wage records for the quarter after exit show earnings greater than zero. UI Wage records will be the primary data source for tracking employment in the quarter after exit. When supplemental data sources are used, individuals should be counted as employed if, in the calendar quarter after exit, they did any work at all as paid employees (i.e., received at least some earnings), worked in their own business, profession, or worked on their own farm.

Exit - Determined as follows:

1. a participant who has a date of case closure, completion or known exit from WIA-funded or non-WIA funded partner service within the quarter (hard exit) or

2. a participant who does not receive any WIA Title I funded or non-WIA funded partner service for 90 days and is not scheduled for future services except follow-up services (soft exit).

Exit Date - the last date on which WIA Title I funded or partner services were received by the individual excluding follow-up services.

Exit Quarter - quarter in which the last date of service (except follow-up services) takes place.

High School Diploma Equivalent - a GED or high school equivalency diploma recognized by the State.

Military service - reporting for active duty.

Not Employed At Registration. An individual who does not meet the definition of employed at registration.

Occupational Skills Goal - primary occupational skills encompass the proficiency to perform actual tasks and technical functions required by certain occupational fields at entry, intermediate or advanced levels. Secondary occupational skills entail familiarity with and use of set-up procedures, safety measures, work-related terminology, record keeping and paperwork formats, tools, equipment and materials, and breakdown and clean-up routines.

Planned Gap in Service - no participant activity of greater than 90 days due to a delay before the beginning of training or a health/medical condition that prevents an individual from participating in services.

Post-Secondary Education - a program at an accredited degree-granting institution that leads to an academic degree (e.g. AA, AS, BA, BS). Does not include programs offered by degree-granting institutions that do not lead to an academic degree.

Qualified Apprenticeship - a program approved and recorded by the ETA/Bureau of Apprenticeship and Training (BAT) or by a recognized State Apprenticeship Agency (i.e., State Apprenticeship Council). Approval is by certified registration or other appropriate written credential.

Response Rate - The percentage of people who have valid contact information who are contacted and respond to the questions on the survey.

Sample - A group of cases selected from a population by a random process where everyone has an equal probability of being selected.

Training Services - Include WIA funded and non-WIA funded partner training services. These services include: occupational skills training, including training for nontraditional employment; on-the-job training; programs that combine workplace training with related instruction, which may include cooperative education programs; training programs operated by the private sector; skill upgrading and retraining; entrepreneurial training; job readiness training; adult education and literacy activities in combination with other training; and customized training conducted with a commitment by an employer or group of employers to employ an individual upon successful completion of the training.

Valid Contact Information - Information that leads the interviewer or surveyor to the location where the contact individual is located whether or not the individual responds and answers the questions.

Work Readiness Skills Goal - Work readiness skills include world of work awareness, labor market knowledge, occupational information, values clarification and personal understanding, career planning and decision making, and job search techniques (resumes, interviews, applications, and follow-up letters). They also encompass survival/daily living skills such as using the phone, telling time, shopping, renting an apartment, opening a bank account, and using public transportation. They also include positive work habits, attitudes, and behaviors such as punctuality, regular attendance, presenting a neat appearance, getting along and working well with others, exhibiting good conduct, following instructions and completing tasks, accepting constructive criticism from supervisors and co-workers, showing initiative and reliability, and assuming the responsibilities involved in maintaining a job. This category also entails developing motivation and adaptability, obtaining effective coping and problem-solving skills, and acquiring an improved self image.

FOOTNOTES
1. The individual would already be registered to be receiving follow-up services.
2. It is planned that outcomes for participants in the Trade Adjustment Assistance and NAFTA Transitional Adjustment Assistance programs will be measured using the same measures described in Measures 5,6 and 7 of this section.
3. Note: The first question can be modified to suit the individual needs of the State and the names for program services recognizable for their population. The lead-in question provided is a model to be used as guidance.
4. DK = Don't Know
5. REF = Refused to Answer
6. This standard is similar to the participant standard that distinguishes core services (information/self-service) from those services that warrant registration.
7. Where an employer has received multiple services in a given time period, and there is separate contact information for each service, the contact information for the priority service should be used for surveying.
8. Note: The first question can be modified to suit the individual needs of the state and the names for program services recognizable for their population. The lead-in question provided is a model to be used as guidance.

Attachment C - Guidance for Calculating WIA Core Measures