¿HAY DIFERENCIAS DE FONDO CON EL CASO DE CHILE?

After the criteria and predictors are chosen, the data collection phase of the validation study can proceed. In this step, measures of the criterion and the predictors are taken on a sample of people to see if the predictor relates to the criterion. A good field test of the predictor is done in the organizational setting in which it is expected to be used in the future. Most validation studies are done in actual organizational settings as opposed to laboratory settings. In the laboratory, you might determine that a human attribute relates to task performance, but you cannot be certain that it will generalize to the organization.

By conducting field studies in the settings in which selection tools will ultimately be used, the likelihood of generalization is maximized.

There are two types of study designs for conducting a validation study. In a concurrent validation study, both the criterion and the predictor scores are collected from a sample of participants at more or less the same point in time. Usually the participants are current employees who are assessed on both criteria and predictors. A sample of employees might be asked to provide predictor data by taking an assessment test. Test scores would then be correlated with employees’ most recent performance evaluation. If the two are related, we assume that scores on the predictor at the time of application for a job will predict later performance on the job.

In a predictive validity study, the predictors are measured before the criterion. A sample of job applicants is assessed on the predictor, but scores on that predictor are not used in deciding who to hire. In other words, we hire applicants who are both high and low on the predictor. Some time (from a few months to a few years) after this group of new hires has been on the job, we assess the criterion or criteria. We conduct statistical analyses to see if the predictor relates significantly to the criterion or criteria.

If the predictor can predict future performance, we can have reasonable confidence in the predictor as a valid selection device.

It might seem that the predictive design would be superior to the concurrent in validating predictors because the predictive design tests the predictor on applicants rather than employees who already have been selected and trained. Because the predictor is used on applicants, generalizability should be maximized. Research has shown, however, that the two designs are equally effective in validating predictors. Validity coefficients, the correlations between scores on the criterion and predictor, have been found to be about the same in studies using the two different types of designs (Schmitt, Gooding, Noe,

& Kirsch, 1984). This is good news for organizations because predictive designs take a long time to conduct. You might have to wait a year after collecting predictor scores to collect criterion scores. Furthermore, for some organizations it could take months or years before a sufficient number of people are hired to conduct the analysis. A concurrent study can be conducted in as little as a few days if the predictor can be administered quickly and the criterion scores are readily available.

Selecting Employees " 147 Step 5: Cross-Validate

The final step in a validation study is to cross-validate or replicate the results of one sample with those of another sample. This is done to be certain that our results are due to a real correlation between the criterion and predictor and not a statistical error. In any study involving statistics, significance can occur by chance as opposed to occurring due to real relations among the variables of interest. Such statistical errors are called Alpha or Type 1 errors. To protect ourselves from making an error in our conclusions about whether or not a predictor can forecast a criterion, we cross-validate or repeat our analyses on another sample of participants. It is extremely unlikely that we will find the same results twice if there is no relation among the variables of interest. In other words, two successive Alpha errors are unlikely.

To conduct a cross-validation, we need two samples. The first sample is used to determine if the criterion and predictor are significantly correlated. A second sample is used to see if the significant relationship found in the first sample can be repeated on the second. The predictor is validated on the first sample and then double-checked or cross-validated on the second. Cross-validation adds to our confidence that the predictor can forecast the criterion or criteria of interest. In most field settings, cross-validation is done by taking the original sample and dividing it randomly in half. The first half is used for the validation, and the second is used for the cross-validation.

Validity Generalization

At times, it is not necessary to collect data to validate a selection test or other assessment device. Selection tests that are valid in one setting are often valid in many other settings.

Validity generalization means that validities of selection devices are generalizable or transportable from job to job and organization to organization (Schmidt & Hunter, 1977).

If a test predicts performance for an administrative assistant in one organization, for example, it will predict for an administrative assistant in another organization.

The idea of validity generalization has been widely accepted among I/O psychologists (Murphy, 2000), at least as long as the jobs and tests in question are comparable. If you validate a test for the selection of people in a particular job, the test should be valid for the same job in a different organization. It should also be valid for a job that has the same KSAO requirements. If the second job is different from the job for which the test was valid, the test in the second case may or may not be valid. The only way to be certain would be to conduct another validation study on the second job to determine if the test predicts the criterion.

How Predictor Information Is Used for Selection

Once it is determined that a predictor or predictors are valid forecasters of future per-formance criteria, it must be decided how best to use the predictor information. Two popular uses of predictor information are as hurdles and as predictors in a regression equation. With either approach, multiple predictors can be used in combination. Often prediction is better with several rather than single predictors because multiple KSAOs are necessary for job success.

Multiple Hurdles

The multiple hurdles approach sets a passing score for each predictor. If an applicant achieves that score, then the hurdle is passed. For example, a computer salesperson should have several KSAOs in order to be successful on the job. One obvious KSAO is knowledge of computer principles. Completion of a college degree in computers could serve as an indicator of the KSAO, and the applicant would pass this hurdle if he or she had such a degree. Another important KSAO might be communication skills so that the person can relate well to customers. This might be assessed with a communication skills exercise. Applicants would have to have a passing score on the communication exercise to pass this hurdle.

It is efficient to use multiple hurdles in a specified order and eliminate applicants as the assessment process goes from hurdle to hurdle. It would make financial sense to order the predictors in terms of cost from least to most expensive. For example, only those computer sales applicants with college degrees would be given the communication skills exercise, since the earning of a degree can be seen in an initial application, whereas the exercise will have some additional cost to administer. Many organizations use relatively inexpensive preliminary screening methods as hurdles so that expensive assessments are not used with people who easily could have been screened out earlier in the process.

Regression Approach

The regression approach uses the score from each predictor in an equation to provide a numerical estimate or forecast of the criterion. With the computer sales job, an equation could predict the actual dollar amount of sales per month. Predictors for that job might be GPA in college and scores on the communication exercise. Both quantitative variables (GPA and exercise score) can be combined mathematically to provide forecasted criterion scores (e.g., monthly sales). Individuals who are forecasted to have the best criterion scores would be those who are hired.

With a single-predictor variable, a linear regression equation is calculated from a sample of data. To compute an equation, you must have data on both the criterion and the predictor so that you can compare how well the forecasted criterion scores match the real criterion scores. The general form of a linear regression equation is

Y = b × X + a

where X is the predictor, Y is the criterion, b is the slope, and a is the intercept. When the equation is used, a and b are known quantities. A forecasted value for the criterion (Y ) can be computed by replacing X with values of the predictor.

The regression equation is developed from the data of a validation study. In addition to the correlation coefficient, a regression equation can be computed for a sample of data on a criterion and predictor. As noted earlier, this equation provides a means of forecasting the criterion from the predictor. For example, monthly sales for a salesperson might be forecasted from scores on the communication exercise. The most accurate forecast might be achieved from a regression equation such as the following:

Sales = $400 × Exercise Score + $2,000

In this equation, a is $2,000 and b is $400. If a person had an exercise score of 10, his or her sales would be predicted to be $6,000:

Sales = $400 × 10 + $2,000 Sales = $6,000

Selecting Employees " 149 If another person had a test score of 5, his or her sales would be predicted to be $4,000:

Sales = $400 × 5 + $2,000 Sales = $4,000

Obviously, the first person would be preferred because his or her forecasted performance is higher.

A similar procedure is applied when there are two or more predictors. This case involves the use of multiple correlation and multiple regression. Multiple correlation is the correlation between a criterion and two or more predictors simultaneously. The multiple correlation coefficient is indicated by an R. Multiple regression is a statistical technique that provides an equation relating two or more predictors simultaneously to a criterion. The equation can be used to forecast the criterion from scores on the predictors.

In many cases, several predictors combined can provide a more accurate forecast of the criterion than any of them alone.

The general form of a multiple regression equation is Y = (b1× X1) + (b2× X2) + a

for the two-predictor case. In this equation, the X s are predictors, Y is the criterion, a is the intercept, and the bs are regression coefficients. The coefficients and intercept are computed from sample data. The equation is solved by substituting values of the predictors for the X s. A forecasted value for the criterion is then computed.

For example, we can combine the scores on the communication exercise with GPA in college. Assume that both of these predictors relate to sales performance. Combined they might provide more-accurate forecasts than either one alone. If each predictor had a correlation of .40 with sales, combined they would likely have a multiple correlation that is greater than .40. The magnitude of the multiple correlation is a function of how strongly each predictor variable correlates with the criterion variable and how strongly the predictor variables correlate with one another. The multiple correlation will have the largest value when the predictor variables are uncorrelated with one another. This would show that combined the predictors are more accurate than either one alone in forecasting the criterion.

A multiple regression analysis would provide an equation that would forecast sales from both the exercise score and the college GPA. The equation could be used to forecast the sales from scores on the two predictors. Suppose that the predictor equation was the following:

Sales = ($2,000 × GPA) + ($1,000 × Exercise) + $2,000

In this equation, a is $2,000, and the bs are $2,000 and $1,000. To use the equation, multiply GPA by $2,000, and add it to the exercise score multiplied by $1,000. To this total add $2,000. The resulting number is an estimate of the person’s future monthly sales.

If a person had a college GPA of 2.0 and an exercise score of 4, his or her forecasted sales would be $10,000. A person with a 4.0 college GPA and an exercise score of 10 would be forecasted to have sales of $20,000.

The magnitude of the relation between the predictors and the criterion determines how accurate the prediction is likely to be. If the predictors correlate strongly with the criterion, the forecasted values for sales are likely to be fairly accurate. If the predictors do not correlate very well with the criterion, the forecasts will not be very accurate. Even

when predictors relate to criteria modestly, however, using the scientific approach we have discussed can still result in hiring better-performing employees than using nonscientific approaches.

Every regression equation must be cross-validated to be sure that it continues to make reasonably accurate forecasts. An equation that is generated from a sample of data will make the most accurate predictions possible for that sample. For statistical reasons that are beyond the scope of this book, it is not likely that the same equation will be as accurate when used on a second sample. To perform a cross-validation, the equation generated from one sample of data is applied to a second sample of data. Usually the accuracy of forecasting will be reduced when using the first sample equation on the second sample. If the regression equation yields nonsignificant results when used on a second sample, it should not be used.

An implication of using the regression approach is that a low score on one predictor can be compensated for by a high score on another. The multiple hurdles approach avoids this problem because an applicant must reach the passing score for each predictor. This can be important because a person often must have a reasonable level on every KSAO even if some KSAOs are very high. For example, in selecting a surgeon there are two equally important KSAOs. He or she must have the knowledge of how to operate and the manual skill to do so. A high level of one KSAO cannot overcome a deficiency in the other. Skill with a scalpel is insufficient if the surgeon does not know where to cut. The limitation of the regression approach can be overcome by combining it with the hurdles approach. First, applicants would be screened using the hurdles. A regression equation would then be applied only to those who made it past the hurdles.

Even when validation studies have been conducted, it is far more common for com-panies to use subjective approaches to combine results of different predictors than to use formal multiple hurdles or regression results, the latter of which is rare in practice.

Ganzach, Kluger, and Klayman (2000) conducted a study that compared the subjective

INTERNATIONAL RESEARCH

Although it has been known for a long time that sub-jective judgment can be less accurate than obsub-jective procedures for combining information to make deci-sions, most organizations use the subjective approach for making hiring decisions. Ganzach, Kluger, and Klayman (2000), in this study of selection in the Israeli Army, wanted to compare these two approaches.

This study was conducted in a field setting in Israel, where military service is compulsory. Partici-pants were 26,197 males who were interviewed prior to being drafted into the army. Each interview took approximately 20 minutes and was conducted by one of 116 highly trained professional interviewers who

underwent a 3-month training program. At the end of the interview, the interviewer made ratings on six traits: activity, pride in the service, sociability, respon-sibility, independence, and promptness. A global rating of expected success in the army of the interviewee was also made. The criterion was the number of dis-ciplinary actions taken against the interviewee during his subsequent 3-year service. Because most partici-pants (83%) had no disciplinary actions taken and few had more than one or two, the criterion was collapsed to two levels: had actions and didn’t have actions.

A multiple regression analysis was conducted between the six trait ratings and the criterion, and

Selecting Employees " 151

a correlation was computed between the global rat-ing and the criterion. Results showed that the six trait ratings, combined with a regression equation to max-imize prediction, was more accurate than the global ratings, with correlations of .28 versus .23, respec-tively. In other words, the statistical combination of the individual trait ratings did a better job predicting the subsequent criterion than the human’s global judg-ment. However, the six trait ratings combined with

the multiple regression equation did even better when used with the global judgment, with a correlation of .30. This suggested that there can be advantages to combining both approaches to achieve maximum pre-diction.

Source: From “Making Decisions From an Interview:

Expert Measurement and Mechanical Combination,” by Y. Ganzach, A. N. Kluger, and N. Klayman, 2000, Personnel Psychology, 53,1–20.

approach with multiple regression for the recruitment of Israeli soldiers. They found that regression was superior in forecasting a measure of performance, suggesting there can be advantages to this approach but that a combination of regression and subjective judgment worked best (see the International Research box).

Alternatives to Conducting Validation Studies

Most organizations select employees without going through costly and time-consuming validation studies. Organizations do not always hire enough people to conduct such studies, which can require more than 100 participants to do properly. Other times orga-nizations do not wish to invest the money or time to conduct these studies. For an organization with hundreds of different jobs, it could cost millions of dollars to conduct validation studies for every position.

An alternative approach is to rely on the established validity of selection tools that can be linked to KSAO requirements. With this approach, one conducts a job analysis to determine KSAOs. Established methods to assess each KSAO are then chosen. If the job analysis results indicate that cognitive ability is needed, an existing cognitive ability test can be chosen. This approach relies heavily on existing research findings concerning the validities of existing methods. It does not involve data collection to test for validity of predictors. An organization can often rely on validity generalization results to help guide its choice of selection methods.

It is possible to purchase existing selection devices that have been developed else-where. Psychological testing companies have validated tests for sale to organizations.

As we discussed in Chapter 5, many tests exist to assess hundreds of different charac-teristics. It is even possible to hire members of consulting firms to administer all sorts of assessments, including assessment centers, interviews, simulation exercises, and tests.

Sometimes it is less expensive for an organization to buy assessment services than to do its own. This is likely to be true with a small company that has few people to assess or a large company that is hiring few people into a particular type of position.

Sometimes it is less expensive for an organization to buy assessment services than to do its own. This is likely to be true with a small company that has few people to assess or a large company that is hiring few people into a particular type of position.