Impuesto Único a la Renta Para la Actividad Bananera

The goal of the SIDES approach is different to that of IT and STIMA in that it aims to detect subgroups of patients with enhanced treatment effect. The IT and STIMA methods grow trees that represent interactions or differential treatment effects that have been detected. The SIDES method also searches for large differential treatment effects however once an optimal split is identified, it retains the subgroup with the enhanced treatment effect and disregards the rest of the sample. The SIDES procedure evaluates all possible splits for every covariate using a splitting criterion that computes the differential effect between the two subgroups formed by a split. The splitting criterion computes a p-value for all searched splits and is of the form

𝑝 = 2 ∙ [1 − Φ (|𝑍𝐸1− 𝑍𝐸2|

√2 )], (6.10)

where 𝑍𝐸1 and 𝑍𝐸2 are the one-sided hypothesis test statistics computed for the two

subgroups respectively and Φ (|𝑍𝐸1−𝑍𝐸2|

√2 ) is the cumulative distribution function of the

standard normal distribution (135). The one sided test statistics are obtained from a simple linear regression model with just the treatment variable as the covariate where

107

the test statistics are computed by dividing the treatment effect estimate by its standard error (t-statistic).

Multiplicity adjustment

Like all recursive partitioning methods, the SIDES algorithm requires all splits to be searched for every covariate at each iteration. A well-known issue with this is that the procedure has a greater probability of selecting a covariate with a larger number of levels. This issue is referred to as variable selection bias (144, 145). Thus to adjust for this, the authors introduced a Sidak-based multiplicity adjustment given by 1 − (1 − 𝑝𝑖)𝐺∗, where 𝑝𝑖 is the unadjusted p-value obtained using the splitting criterion

(equation 6.10) for the i-th split and 𝐺∗ _{is the effective number of splits (135). The}

effective number of splits basically incorporates the correlation between the p-values for any single covariate and is estimated by 𝐺∗_{= 𝐺}1−𝑟̅_{, where 𝐺 is the number of splits}

for a given covariate and 𝑟̅ is the mean correlation of the p-values computed using the splitting criterion having evaluated all splits. Thus for a given covariate, the associated p-values for all potential splits are adjusted to reduce the variable selection bias.

Complexity control

The complexity of a tree simply refers to the size of a tree usually defined by the number of terminal nodes it has. Both the IT and STIMA approaches use pruning and cross-validation to control for the complexity of a tree and thus select the best tree size respectively. The SIDES approach on the other hand controls the complexity of a tree during the tree growing process by using a relative improvement parameter at each level of the tree. The relative improvement parameter determines whether or not a child node with a large positive treatment effect should become a parent node for the next iteration of the algorithm. A child node only becomes a parent node if there is

108

some improvement in the p-value of the child node compared to the p-value of the parent node from which it came from. The split is made if

𝑝_𝑐 ≤ 𝛾 ∙ 𝑝_𝑝, (6.11) where 𝑝𝑐 is the p-value of the child node, 𝑝𝑝 is the p-value of the parent node and 𝛾 is

the relative improvement parameter which ranges from 0 to 1. A small value of 𝛾

makes the procedure much more selective; only selecting subgroups with a very small p-value. Conversely, a value of 1 for 𝛾 is less restrictive and thus allows the procedure to search a wider covariate space i.e. build a bigger tree.

Each level of the SIDES tree has an associated relative improvement parameter. For example, if some restriction was only required for the first three levels of the tree, then one would need to specify 𝛾_𝑖 for stage i. The relative improvement parameters can be user specified. If none are specified, they can be determined using 5-fold cross- validation to try and find the optimal combination. The cross-validation procedure is implemented in the same way as described in section 6.2.5. This requires the user to define the grid space that needs to be searched for the optimal combination of the 𝛾

parameters. For example, say we want to define a subgroup using three covariates only i.e. a tree with three levels, then we need to specify the values for 𝛾₁, 𝛾₂ and 𝛾₃ to define the grid that needs to be searched. Let’s say we specify that 𝛾₁ goes from 0.1 to 1.0 in increments of 0.1 and both 𝛾₂ and 𝛾₃ go from 0 to 1 in increments of 0.1, then the grid is formed using all combinations of 𝛾1, 𝛾2 and 𝛾3. Thus in this example, the first

combination would be 𝛾 = (0.1,0,0) and the last combination would be 𝛾 = (1,1,1). Therefore, the 5-fold cross-validation works by applying the SIDES procedure to the training sample for each combination of 𝛾 in the grid. The best subgroup with the smallest treatment effect p-value is then identified and the same subgroup evaluated in the test sample and the corresponding p-value recorded. This is then repeated five

109

times for each fold in the cross-validation. Finally, the p-values are averaged across the 5-folds for each combination in the grid and the combination with the smallest average p-value is chosen as the optimal combination to be used by SIDES. Though this

approach is good for finding the optimal combination, it can be rather time consuming if there are a large number of levels for which a relative improvement parameter is required. Therefore, the authors recommend only searching for the optimal

combination for the first three levels of the tree.

Growing a tree – parameter specifications

Now that the splitting criterion, multiplicity adjustment and complexity control have been defined, the algorithm for growing a SIDES tree will now be described. The authors very kindly provided me with the coding for the SIDES method written in R software. When applying the SIDES procedure, it is required that a number of parameters be specified to aid the tree growing process. Like the IT and STIMA methods, SIDES also requires that the minimum node size and maximum number of levels of a tree be pre-specified. Specifying the number of levels of a tree helps restrain the complexity of the subgroups. The number of levels is equivalent to the number of covariates that can define a subgroup. For example, if the maximum number of levels is set to 3, then a subgroup can, at most, be defined by 3 covariates. However, in addition, SIDES also requires the pre-specification of the best number of splits, say M, to be considered for each parent node at each level. For example, if M=3, then for any node, all splits will be evaluated for each covariate retaining only the single best split for each covariate. The best splits are then ordered from smallest to largest and the best three covariate splits are chosen to split on. The SIDES procedure also requires the user to either specify the relative improvement parameters to be used, or specify the grid space to be searched for the optimal relative improvement parameter combination

110

using cross-validation as described earlier. Once the aforementioned parameters have been pre-specified, the SIDES procedure algorithm can be applied as follows

 Initialise: Start at the root node (level 0) consisting of the entire dataset

 Iteration:

o Step 1 - Evaluate the splitting criterion for all splits of every covariate (exclude covariates already used to define the parent node) retaining only the best split for each covariate. Order the covariates from smallest adjusted p-value to largest adjusted p-value where the adjusted p- values are computed using the Sidak-based multiplicity adjustment. Ordering the adjusted p-values ensures that covariates with a larger number of splits are not favoured i.e. not placed higher up in the ordering due to multiplicity.

o Step 2 - Select the best M covariates from the ordered best splits. For each of the M splits, form the split creating two child nodes and retain the child node with the larger positive treatment effect, provided it satisfies the relative improvement parameter condition (see equation 6.11). The retained nodes now become parent nodes for the next iteration.

o Step 3 – Repeat steps 1 and 2 for the newly formed parent nodes

o Step 4 – Continue the above three steps until either the maximum number of levels is reached or if no more splits can be formed i.e. relative improvement parameter condition not satisfied. In both cases, the previously formed parent nodes become terminal nodes.

The above algorithm is implemented until a SIDES tree has been grown. The resultant tree is a collection of candidate subgroups that all have an enhanced treatment effect

111

where each subgroup is defined by up to M covariates. However, many of the candidate subgroups might be spurious findings due to inflated type I error rates. Thus to control for this and to remove possibly spurious subgroups, the authors propose a resampling based procedure to adjust the p-value for each candidate subgroup and control the type I error in the weak sense (135). The resampling procedure firstly randomly permutes the rows of the outcome variable and treatment variable in the dataset to form a null dataset. The outcome and treatment variables are kept together and permuted then reattached to the covariate values. It is done in this way to ensure the overall treatment effect is maintained as well as the correlation structure of the covariates. The SIDES procedure is then applied to the null dataset using the same parameters used to grow the initial tree i.e. the same complexity parameters, minimum node size, etc, and the p-value of the best subgroup recorded. This process is repeated many times, e.g. 500 or 1000 times, forming a distribution of p-values. The observed p- values from each of the candidate subgroups is then adjusted by calculating the proportion of p-values in the distribution from the resampling procedure that fall below the observed p-value. More precisely, the adjusted p-value for a given candidate subgroup is computed by

𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑 𝑝 − 𝑣𝑎𝑙𝑢𝑒 =number of p − values from null less than observed p − value

number of permutations .

Once the p-values for each of the candidate subgroups have been adjusted using the resampling procedure, inferences can be made by observing the adjusted p-values and thus the final subgroups chosen. This basically means that the adjusted p-value is a comparator for the unadjusted p-value in the sense that it reflects how true the subgroup found may be. If the unadjusted p-value is significant and the adjusted p- value is non-significant, then this suggests that the identified subgroup is quite possibly a spurious finding.

112

SIDES algorithm illustration

A simple example will now be considered to better illustrate how the SIDES procedure works; see Figure 6.3. Assuming we have three covariates, say X1, X2 and X3, and that

we allow SIDES to split up to a maximum of two levels i.e. any candidate subgroups found can be defined by up to two covariates. Moreover, assume that we select the best two splits (M=2) at each level of the tree. The SIDES procedure initially starts at level 0 with a single parent node consisting of the entire dataset, illustrated by a black circle. From the first iteration, the covariates X1 and X2 contain the best two splits at split

points S1 and S2 respectively. The treatment effect in subgroup X1≤S1 is substantially

larger than the effect in X1>S1, thus the sample X1>S1 is disregarded and the sample

with X1≤S1 is retained thus forming a parent node at level 1. Similarly for the second

split on X2, the sample X2>S2 is retained and the sample X2≤S2 is disregarded thus the

sample X2>S2 forms another parent node at level 1. Any disregarded nodes after a split

has been made have been coloured grey in figure 6.3. Therefore at level 1, there are two new parent nodes (two black circles) that can both be searched for the best two splits respectively in the next iteration. Note that only those covariates that have not already been used to define the parent node are searched for the next split. For

example, if at level 1 the covariate X1≤S1 is used to define the parent node, then the next

iteration of SIDES can only search covariates X2 and X3 for the next best two splits.

Exactly the same process is carried out on the two parent nodes at level 1. The first parent node at level 1 can be split by the covariates X2 and X3 at split points S12 and S13

respectively. From these splits, X2≤S12 and X3≤S13 are retained to form two new parent

nodes at level 2. Similarly, the second parent node at level 1 can be split by the covariates X1 and X3 at split points S21 and S23 respectively. From these splits, X1≤S21

and X3≤S23 are retained to form two more parent nodes at level 2. Thus there are four

new parent nodes formed at level 2. If we recall earlier, we specified the maximum number of covariates used to define a subgroup as being two i.e. maximum of two

113

levels. Thus, the SIDES procedure stops here and the four parent nodes identified at level 2 thus become terminal nodes. Therefore at the end of the procedure, four candidate subgroups have been identified; subgroup1 {X1≤S1 and X2≤S12}; subgroup2

{X1≤S1 and X3≤S13}; subgroup3 {X2>S2 and X1≤S21}; subgroup4 {X2>S2 and X3≤S23}.

Though this example has been illustrated using Figure 6.3 in the form of a single tree (i.e. in the same form illustrated by the authors) it can also be illustrated as two separate trees defined by the two branches stemming from the root node. The reason we can do this is because each branch stemming from the root node is essentially a different way in which we can initiate the tree growing process starting with the entire dataset. Thus, if we specify that we want the procedure to consider the best M splits for each node, then this means that we can illustrate the final identified subgroup by up to M separate trees with the root node being the entire dataset.

6.4 Discussion

Tree based methods are a promising alternative to performing exploratory subgroup analyses in randomised controlled trials. These methods overcome many of the issues with existing conventional subgroup analyses as highlighted in previous chapters. Moreover, all tree based methods utilise a simple technique referred to as recursive partitioning. Therefore, this chapter provided a detailed description of the recursive partitioning methodology to better understand the method. The method was described in reference to the commonly used and well established CART type procedure simply to demonstrate how the method works. The CART method is typically applied to data where the goal is to detect interactions that are predictive of outcome; however it does not detect treatment effect heterogeneity i.e. treatment-covariate interactions; which is the focus of this thesis. However, there are a number of advanced variants based on the CART type procedure, namely the IT, STIMA and SIDES methods, which do look for treatment effect heterogeneity. These methods were thus described with reference to

114

the recursive partitioning methodology described earlier on in the chapter since the steps in the algorithm of these methods are relatively similar.

This chapter highlighted a clear difference in the aims of the IT and STIMA methods compared to the SIDES method. The IT and STIMA methods both look to identify subgroups with the aim of maximising the treatment-covariate interaction effect whereas the SIDES method aims to detect subgroups of individuals who have an enhanced treatment effect. Despite there being this difference, from a clinical perspective, both aims are very important.

This chapter gave a detailed insight into tree based methods and the underlying methodology. Moreover, three recently proposed advanced variants for performing subgroup analyses or subgroup identification were also described. Having identified and described these methods, it is important to evaluate them in a variety of simulated scenarios to assess if they actually do what they aim to do. Therefore, the next chapter will describe a simulation study to evaluate these methods with the aim of identifying the best method(s) for performing subgroup analyses or subgroup identification.

115

Figure 6.3 – Example of the SIDES procedure with two levels

X1≤s1 X1>s1 X2≤s2 X2>s2 X2≤s12 X₂>s₁₂ X3≤s23 X3>s23

LEVEL 1

LEVEL 2

LEVEL 0

116

Chapter 7

Simulation study to

evaluate tree based

methods

7.1Introduction

The previous chapter introduced the recursive partitioning methodology; a key component of tree based approaches. Although there are several variants of the tree based method that employ a CART type routine, there are only a few that specifically look to identify differential subgroups effects, which is what is of interest in this PhD. Moreover, these methods enable us to identify subgroups defined by multiple baseline characteristics. The tree based methods that do identify differential subgroup effects are interaction trees (IT), Simultaneous Threshold Interaction Modelling Algorithm (STIMA) and Subgroup Identification based on a Differential Effect Search (SIDES) (133, 135, 136). As highlighted in the previous chapter, there is a clear distinction in the aims of both the IT and STIMA methods compared to the SIDES method. The IT and STIMA methods aim to identify baseline characteristics that are moderators of

117

whereas the SIDES method aims to identify subgroups with enhanced treatment effect i.e. each split in the tree represents an identified subpopulation with a large treatment effect.

Both of the aforementioned aims from a clinical perspective are very important. Therefore, all three of these methods stand as good candidate methods that could potentially be extended to an IPD subgroup meta-analysis setting to identify subgroups defined by multiple characteristics. Before considering ways to extend these methods, it is initially important to evaluate how well these methods perform in a single trial setting. This chapter will therefore describe the setup of the simulation study in a