Modelación de problemas ingenieriles

To recap, the previous sections in this chapter have so far introduced the IPD meta- analyses framework and discussed the statistical issues with applying conventional analyses in this setting. Moreover, statistical approaches currently used to better deal

145

with the clustering of individuals within studies, in particular the one-stage fixed- effects and mixed-effects modelling approaches, were introduced. Specifically, a number of simple fixed-effect and mixed models were described that can be used to perform subgroup analyses; fixed intercept model, random intercept model and the random intercept and random slope model.

The objective of this section of the chapter is to propose extensions to the IT and SIDES methods such that they can be applied to IPD to perform subgroup analyses. A natural approach to consider is to somehow incorporate the currently used fixed-effects and mixed-effects models into the tree based procedures to ensure they account for the clustering within studies. A recent critical review of statistical methods for detecting interactions in IPD meta-analyses by Fisher et al proposed a model that was considered to be the most basic yet useful for a one-stage approach. This model includes a vector of study indicator variables i.e. 𝛽_0𝑖 (fixed effects) with fixed covariate and interaction effects i.e. 𝛽₂ and 𝛽₃ respectively have fixed effects, and with the treatment effect either being fixed, as shown in equation (8.1), or random, as shown in equation (8.3) (162). The authors also note that any of the components in this basic model can potentially be specified as being either fixed or random. Therefore, a number of variations of this basic model can be considered i.e. varying the fixed and random effects components, when considering possible extensions. At this point, let’s assume that the intercept can be either fixed or random and that the treatment, covariate and the interaction are the same for all trials i.e. have fixed effects. Hence there are three basic model variations (including a null model) that can be considered

 Model A – a null model that ignores the clustering i.e. a general linear model that does not include fixed or random trial effects

146

 Model B – a model with fixed trial effect, fixed treatment effect and fixed interaction effect (Fixed-effects model) – as shown in equation (8.1)

 Model C – a model with random trial effect, fixed treatment effect and fixed interaction effect (Mixed-effects model) – as shown in equation (8.2)

Having discussed the possible basic models that can be considered, we then need to think about how they can be incorporated into extending the tree methods for

application to IPD subgroup analyses. It is important to note at this stage that the aim of the extended IT and SIDES methods are not to obtain some pooled effect size estimate e.g. treatment effect or interaction effect, which is typically the aim of most IPD meta-analyses. Rather, the aim will be to identify subgroups of individuals defined by multiple characteristics that either maximize the differential treatment effect (as done by IT) or have enhanced treatment effect (as done by SIDES).

IT method extension (IPD-IT)

Recall that both IT and SIDES use recursive partitioning which heavily relies on a splitting criterion. Therefore, in order for both IT and SIDES to be extended to an IPD setting, they require a new splitting criterion to be defined. Let us first consider the extension of the IT method. In a single trial setting, the IT procedure used a splitting criterion based on the t-test statistic (Chapter 6 – equation (6.8)) to test the null hypothesis that the coefficient of the interaction effect is equal to zero. Therefore, a natural extension to an IPD meta-analysis setting is to use a splitting criterion based on the t-test statistic for the interaction effect when fitting a fixed-effects or mixed-effects model. The model would have accounted for the correlation within studies and so the estimate of the standard error for the interaction effect would be reliable; hence the estimate of the t-test statistic will be reliable. The t-test statistic is computed by

147

𝑡(𝑠) = 𝛽̂3 𝜎_𝛽3

where 𝛽̂3 is the estimated coefficient of the interaction term and 𝜎𝛽3is the associated

estimated standard error. The computed statistic can either be positive or negative, indicating the direction of the interaction. The tree growing procedure aims to maximize the interaction effect at each level of the tree regardless of the direction of the interaction. For example, it could either be a large positive interaction or a large negative interaction. Hence a splitting criterion is required that focuses on maximizing the absolute interaction effect. Thus, the splitting criterion can be defined as

𝐺(𝑠) = 𝑡2_(𝑠)

which is analogous to the splitting criterion used for the IT procedure in a single trial

scenario. The criterion 𝐺(𝑠) is essentially the Wald test statistic i.e. 𝛽̂32

𝜎_𝛽32 to test the null

hypothesis that the interaction effect is equal to zero. Therefore, during the tree growing process, an optimal split is defined as the split that maximizes 𝐺(𝑠). The splitting criterion can be estimated using either a fixed-effects model (model B) or a mixed-effects model (model C). Note here that the splitting criterion associated with model A is computed in the same way as the original IT method for a single trial. What we now require is to choose which out of these three models (A, B and C) provides the best estimate of 𝐺(𝑠) having accounted for the hierarchical data structure so that the IT method grows the correct size tree. This decision depends on what you believe about the distribution of the data. Therefore, it would be good to compare the three models as estimators of 𝐺(𝑠) to see how badly the simpler models do when the data really come from a more complex model. This will be evaluated in the next section of this chapter by performing a simulation study. From now onwards, the extended IT method

148

will be referred to as IPD-IT, which refers to three methods corresponding to models A, B, and C above.

SIDES extension (IPD-SIDES)

A new proposed splitting criterion for the IT method in an IPD meta-analysis setting has now been defined (IPD-IT). Let us now consider the extension of the SIDES method to an IPD framework. Recall that the SIDES procedure in a single trial setting simply computes the one-sided t-test statistic for treatment effect in both the child nodes separately. The procedure also stores the associated p-value from the one-sided test for inferential purposes should the node be chosen to define a subgroup. Thereafter, these two statistics are fed into the splitting criterion (Chapter 6 – equation (6.10)), which is essentially a test for interaction, to compute a p-value to test the absolute value of treatment effect heterogeneity between the two nodes. Due to asymptotics, the t-test statistics are treated as z-statistics and thus the splitting criterion computes a p- value assuming the test statistics have a standard normal distribution. Hence, one can easily extend the SIDES method by computing the p-value in a similar manner using the same splitting criterion but substituting in the treatment effect t-test statistic

𝑡(𝑠) = 𝛽̂1 𝜎_𝛽1

from either the fixed-effect model (Model B) or the mixed-effect model (Model C), where 𝛽̂1 is the treatment effect estimate and 𝜎𝛽1is the associated standard error.

Though we can obtain p-values for the one-sided tests for treatment effect in a fixed- effects model, it is not that simple when using a mixed-effects model. The reason for this is that in a hierarchical setting, it becomes difficult to count the degrees of freedom which are required to set the distribution for the test statistic. If the variance of the fixed-effects component is known then one can easily obtain a p-value. However, the

149

variance is not known and is estimated using REML, hence there will be some degree of uncertainty surrounding the variance estimates. As a result, the cumulative

distribution function of the test statistic will be unknown and thus a p-value cannot be calculated (157, 160). A number of approaches have been proposed to better

approximate the degrees of freedom and thus obtain an approximate p-value (163, 164). However there is still much debate about how to best approximate the p-values and so there is no simple solution. Despite the ongoing debate, if a p-value is required, a sensible distributional assumption for the test statistic can be made based on the data and thus an approximate p-value obtained. More specifically in the context of IPD, due to the large amounts of data, one can treat the t-test statistic from the model as a z- statistic or z-score. Hence, a p-value from a one-sided test can then be estimated using a standard normal distribution which doesn’t require the degrees of freedom to be specified. Thereafter, the one-sided z-statistics can be substituted into the splitting criterion to estimate the p-value of the interaction effect to aid the SIDES tree growing procedure. It is worth highlighting here that the p-values are not estimated to test hypotheses; rather they are utilized by the SIDES procedure to aid the exploratory search process. This therefore further justifies the distributional assumptions made for the test statistics in the fixed component of the mixed model. From now onwards, the extended SIDES method will be referred to as IPD-SIDES, which refers to three methods corresponding to models A, B and C above.

Now that splitting criteria have been defined for the IPD-IT and IPD-SIDES methods, what we now require is to determine which of the splitting criteria associated with models A, B and C provides reliable estimates of the splitting criterion score in order to detect interaction effects or identify subgroups with large treatment effects. The choice of model and thus the associated splitting criterion can be determined by performing a

150

simulation study considering different subgroup scenarios as well as varying degrees of between-study heterogeneity. The simulation study setup will now be described.

8.5Simulation study setup Data generation procedure

The simulation setup to evaluate the IPD-IT and IPD-SIDES method required that data be generated for several studies thus forming a pooled dataset. The same simulation setup described in Chapter 7 (section 7.2) was used to generate data for a single trial. Recall from Chapter 7 that the data generation process simply requires the main effects and interaction effects to be specified in order to generate the data. The same data generation procedure was applied several times to generate multiple single trial data that were then combined to form a pooled IPD dataset. However, in addition, this required some between-study variation, typically denoted by 𝜏2 _{in meta-analyses, to be}

introduced to represent the clustering present in the hierarchical data structure. This was done by randomly generating an overall mean 𝛽₀ for each trial in the pooled dataset from a normal distribution 𝛽0~N(0, 𝜏2) with a mean of zero and a between-

trial variance of 𝜏2_{, where 𝜏}2_{is to be specified. It is probably worth noting here that the}

ratio of the between-study variation 𝜏2 _{over the total variation provides another}

measure of heterogeneity commonly used in meta-analyses referred to as the

intraclass correlation (ICC). Hence, the ICC measures how much of the total variance is explained by the clustering within each study. Values of the ICC typically range from zero to one where a value of zero suggests that the response values in one study are similar to those in other studies and a value of one suggests they are completely different.

151

Simulation study design

A full factorial simulation study design was used where four factors were varied; the sample size of each study in the pooled dataset, the interaction effect size for 𝑇 ∙ 𝑋₁, the interaction effect size for 𝑇 ∙ 𝑋₂ and the between-study variance. Each pooled dataset consisted of 5 studies with each study having a sample size of 200, 500 and 1000. For simplicity, the sample size of each study in the generated pooled dataset remained fixed. Furthermore, the simulated data assumed an equal proportion of individuals in each category of the treatment variable 𝑇, and the two covariates 𝑋₁ and 𝑋₂.

Standardized interaction effect sizes of 0, 0.2 (small), 0.5 (medium), 0.8 (large) and 1.5 (very large) were considered for both the 𝑇 ∙ 𝑋₁ and 𝑇 ∙ 𝑋₂ interactions. Between-study variances (𝜏2_{) of 0.1 (small) and 0.9 (large) relative to a residual within study variance}

of 1 were considered as they are within range of the typical between-study

heterogeneity found in IPD meta-analyses (156, 165). These small and large between- study variances equate to ICC values of approximately 0.08 and 0.42 respectively in the simulated datasets. Varying 𝜏2 _{will enable us to investigate and contrast how the}

methods perform in varying degrees of between-study heterogeneity. In total, each permutation in the full-factorial design was simulated 1000 times and the results summarized. The simulations were performed using R software 3.0.1.

8.6Simulation study results