Otras aplicaciones 36

One important goal in medical studies is prognostic modelling. Prognostic models serve to make predictions about a future outcome of an individual based on predictors in the model. Predictors are variables that should have been identified during the fitting model procedure to play a role on the outcome of individuals. They should have been identified by analysing a representative sample from the population, whose individual’s outcomes are to be predicted. During the fitting of a model these variables are called ”covariates” or ”risk factors”. In the context of prognostic models, they are called ”predictors”.

A fitted model may prove to be a good fit for the data sample, this is determined in the evaluation of goodness of fit of the model. However, to prove if a model is a good prognostic model, it should be validated. Validation of a model is done through the estimation of prediction errors. By estimating prediction errors we test the fitted model to new data, independent from the data sample used in the fitting procedure. By validating a model we prove that the risk factors predict well not only the outcomes for the particular individuals in a cohort, but also the outcome for any other individual from the study population. In genetic association studies, validated genetic models can be used to predict the risk of disease of an individual patient based on the genotypes carried at the specific loci in the genetic model. It can also be useful to predict the risk of a patient undergoing a specific therapy based on a joint influence of the genotypes with clinical or environmental predictors.

Validation procedures in Cox regression models has not been widely investigated yet. One approach was suggested by Gerds and Schumacher (2007), who used the criterion of the Brier score to estimate prediction errors. In the context of genetic association studies, we want to focus on the investigation of prognostic Cox models with SNP factors. On this matter, the results of the study by M¨uller et al. (2008) suggest that the criterion of the Schoenfeld residuals could be an acceptable ap- proach. Hence, we developed an approach to estimate prediction errors with the criterion of the Schoenfeld residuals.

The methodologies for estimators of prediction errors, as well as our approach for application on Cox regression models using the criterion of the Schoenfeld residuals, are described in the next chapter.

Prediction error estimators

Model fitting is useful for understanding the information from a population based on some available representative data of that population. By fitting a model we wish to identify variables associated with an outcome based on the information provided by individuals in the data. A fitted model can be a good representation of that information in the data, but that is not a guarantee that the model will represent well the information of individuals who are not in the data.

Model validation is the step to judge if a model will fit well the data of individuals in the population. Hence, we evaluate the validity of the model for a population and not only for the data at hand. A model is valid if it can closely predict the outcome of the individuals based on independent variables in the model, these variables are called predictors.

Through prediction errors we can evaluate the performance of a model with respect to predictions. A good model for prediction is expected to produce low prediction errors. The prediction errors are computed as the squared difference between a future response and its prediction from the fitted model. In practice, the future response can be taken from new data independent from the data used for fitting the model. However, there are not always new data available. Then, some techniques have been proposed to remedy the lack of new data.

Among some proposed techniques to replace the unavailability of data to estimate prediction errors are cross-validation (Stone 1974), bootstrap (Efron and Tibshirani 1993), the 0.632 estimator (Efron 1983), and the 0.632+ _{estimator (Efron and}

Tibshirani 1997). Both the cross-validation and the bootstrap estimator estimate prediction errors by using, in different ways, the available data to repeatedly sim- ulate two subsets: the training sample to fit the model and the validation set to validate the model. The 0.632 estimator and the 0.632+ _{estimator were introduced}

as improvements to the two first aforementioned estimators.

Gerds and Schumacher (2007) extended the applicability of these estimators of pre- diction errors to survival data. Based on the definition of the Brier score (Brier 1950),

they computed prediction errors as the squared difference between the true survival status (taken from the validation set) and the estimated survival probabilities under the model (fitted with the training sample).

We propose the applicability of these estimators of prediction errors to survival data by using the definition of the Schoenfeld residuals (Schoenfeld 1982). By adapting the Schoenfeld residuals to prediction errors, we use the squared difference between the true value of a covariate (taken from the validation set) and the expected value of the covariate under the model (fitted with the training sample). We propose this approach to be applied on genetic association studies, because according to the in- vestigation on genetic association studies by M¨uller et al. (2008), the criterion of the Schoenfeld residuals was the best to evaluate the goodness of fit of a genetic survival model (see section 3.5.4). Hence, we wish to provide the approach of estimation of prediction errors with the criterion of the Schoenfeld residuals.

Our main interest is to evaluate the improvement in prediction of a genetic model with respect to a reference model, e.g. model without the genetic factors. We can use an R2 _{estimator as the goodness of fit but based on estimators of prediction errors.}

It will provide the fraction of contribution of the genetic factors to the prediction of the survival outcome, i.e. the gain in prediction by considering the genetic variables as risk factors for the survival outcome, in comparison to a reference model.

In the first section, this chapter introduces the concept of prediction errors, describes the existing estimators, and presents the adaptation of these estimators to survival data based on the Brier score as developed by Gerds and Schumacher (2007). The second section presents our development, the adaptation of the estimators of predic- tion errors to survival data based on the Schoenfeld residuals. Finally, we formulate the R2 _{estimator based on estimators of prediction errors.}

4.1

Prediction error

The prediction error is a measure of the capability of a model to predict the future response of new observations. In linear regression models, prediction error is defined as the squared difference between the value of a future response and its prediction under the model.

Brier (1950) introduced the concept of verification score to estimate the error of misclassification of the model to a set of categories. For a particular set of two cate- gories, the verification score was defined as the mean squared difference between the occurrence of an event and the estimated probability of occurrence of that event. They used as an example the dichotomous outcome for the event of rain, the occur- rence of rain was coded as rain=1 or no-rain=0, and the probability of raining was estimated from previous climatological observations. In that context, the verification score is equivalent to the above definition of prediction error, here the occurrence of rain (1/0) was the future response and the estimated probability of raining was

the predicted value. The verification score from Brier (1950) is known as the Brier score.

In any case, estimating prediction errors requires a prediction rule to predict the future outcome, and a set of independent new observations, which is called validation set, to test the performance of the prediction rule. The prediction rule can be either a model, a function or any other algorithm constructed with a training sample. Estimating prediction errors is straightforward for models such as the linear regres- sion models since the predicted outcome is obtained directly from the model, that is a linear function of the predictors and the estimated parameters of the model, i.e. the prediction rule is the model itself.

For models such as the logistic regression model, the predicted binary outcome is estimated by a probability π that is a non-linear function of the predictors and the parameters θ = (α, β) of the model (see section 2.2.5), i.e. the prediction rule is not the model itself but a function of its parameters.

The expected prediction error of a prediction rule is formulated as

Err = E { Y0− r (X0) }2, (4.1)

where Y0 and X0 = (X01, . . . , X0K), are the values of the outcome and K predic-

tors, respectively, of a random individual 0 drawn from the study population. r is the prediction rule for the outcome Y0. r(X0) is the predicted outcome given the

predictor X0.

A value of Err = 0 indicates the full validity of the prediction rule, that occurs if Y0 = r(X0). The larger the distance between Y0 and r(X0) the larger the prediction

errors, and therefore the less valid the prediction rule.

In practice we can estimate Err in terms of a prediction error rate by averaging the prediction errors for all individuals in a validation set. Let V0 be a validation set,

and nV0 be the size of V0, then

d Err = 1 nV0 n_V0 X i=1 { Yi− r (Xi) }2. (4.2)

Then, estimating Err requires two independent sets of data: the training sample to estimate the prediction rule r, and the validation set providing the outcomes Yi

and predictors Xi = (Xi1, . . . , XiK), for i = 1, . . . , nV0, to test the performance of the prediction rule. The estimate dErr is the average squared difference between the observed outcome Yi and the predicted outcome r(Xi) over all individuals in the

validation set. The predicted outcome is obtained from the prediction rule given the predictor set Xi.

However, a new data set as validation set is not always available, or to collect new data will be costly or it will require great investment of time. Thus, some tech-

niques have been suggested to estimate Err without demanding new data. These techniques estimate Err using the available data in different ways, by considering the availability of a training sample to estimate the prediction rule, and of a valida- tion set to evaluate the prediction rule. The techniques that we will consider here are: the apparent error (err), the bootstrap error ( dErr_B), the cross-validation error ( dErr_CV), the bootstrap cross-validation ( dErr_B0), the 0.632 estimator ( dErr.632),

and the 0.632+ _{estimator ( dErr.632}+), which are described in the following sections.