Decision trees are a simple model type: they make a prediction that is piecewise con- stant. This is interesting because the null hypothesis that we’re trying to outperform is often a single constant for the whole dataset, so we can view a decision tree as a proce- dure to split the training data into pieces and use a simple memorized constant on each piece. Decision trees (especially a type called classification and regression trees, or
CART) can be used to quickly predict either categorical or numeric outcomes. The best
way to grasp the concept of decision trees is to think of them as machine-generated business rules.
FITTING A DECISION TREE MODEL
Building a decision tree involves proposing many possible data cuts and then choosing best cuts based on simultaneous competing criteria of predictive power, cross-validation strength, and interaction with other chosen cuts. One of the advantages of using a
canned package for decision tree work is not having to worry about tree construction details. Let’s start by building a decision tree model for churn. The simplest way to call
rpart() is to just give it a list of variables and see what happens (rpart(), unlike many
R modeling techniques, has built-in code for dealing with missing values).
> library('rpart') > fV <- paste(outcome,'>0 ~ ', paste(c(catVars,numericVars),collapse=' + '),sep='') > tmodel <- rpart(fV,data=dTrain) > print(calcAUC(predict(tmodel,newdata=dTrain),dTrain[,outcome])) [1] 0.9241265 > print(calcAUC(predict(tmodel,newdata=dTest),dTest[,outcome])) [1] 0.5266172 > print(calcAUC(predict(tmodel,newdata=dCal),dCal[,outcome])) [1] 0.5126917
What we get is pretty much a disaster. The model looks way too good to believe on the training data (which it has merely memorized, negating its usefulness) and not as good as our best single-variable models on withheld calibration and test data. A cou- ple of possible sources of the failure are that we have categorical variables with very many levels, and we have a lot more NAs/missing data than rpart()’s surrogate value strategy was designed for. What we can do to work around this is fit on our repro- cessed variables, which hide the categorical levels (replacing them with numeric pre- dictions), and remove NAs (treating them as just another level).
> tVars <- paste('pred',c(catVars,numericVars),sep='')
> fV2 <- paste(outcome,'>0 ~ ',paste(tVars,collapse=' + '),sep='') > tmodel <- rpart(fV2,data=dTrain) > print(calcAUC(predict(tmodel,newdata=dTrain),dTrain[,outcome])) [1] 0.928669 > print(calcAUC(predict(tmodel,newdata=dTest),dTest[,outcome])) [1] 0.5390648 > print(calcAUC(predict(tmodel,newdata=dCal),dCal[,outcome])) [1] 0.5384152
This result is about the same (also bad). So our next suspicion is that the overfitting is because our model is too complicated. To control rpart() model complexity, we need to monkey a bit with the controls. We pass in an extra argument, rpart.control (use help('rpart') for some details on this control), that changes the decision tree selection strategy.
> tmodel <- rpart(fV2,data=dTrain,
control=rpart.control(cp=0.001,minsplit=1000, minbucket=1000,maxdepth=5)
)
Listing 6.13 Building a bad decision tree
Listing 6.14 Building another bad decision tree
129
Building models using many variables
> print(calcAUC(predict(tmodel,newdata=dTrain),dTrain[,outcome])) [1] 0.9421195 > print(calcAUC(predict(tmodel,newdata=dTest),dTest[,outcome])) [1] 0.5794633 > print(calcAUC(predict(tmodel,newdata=dCal),dCal[,outcome])) [1] 0.547967
This is a very small improvement. We can waste a lot of time trying variations of the
rpart() controls. The best guess is that this dataset is unsuitable for decision trees
and a method that deals better with overfitting issues is needed—such as random for- ests, which we’ll demonstrate in chapter 9. The best result we could get for this dataset using decision trees was from using our selected variables (instead of all transformed variables).
f <- paste(outcome,'>0 ~ ',paste(selVars,collapse=' + '),sep='') > tmodel <- rpart(f,data=dTrain, control=rpart.control(cp=0.001,minsplit=1000, minbucket=1000,maxdepth=5) ) > print(calcAUC(predict(tmodel,newdata=dTrain),dTrain[,outcome])) [1] 0.6906852 > print(calcAUC(predict(tmodel,newdata=dTest),dTest[,outcome])) [1] 0.6843595 > print(calcAUC(predict(tmodel,newdata=dCal),dCal[,outcome])) [1] 0.6669301
These AUCs aren’t great (they’re not near 1.0 or even particularly near the winning
team’s 0.76), but they are significantly better than any of the AUCs we saw from single-
variable models when checked on non-training data. So we’ve finally built a legitimate multiple-variable model.
To tune rpart we suggest, in addition to trying variable selection (which is an odd thing to combine with decision tree methods), following the rpart documentation in trying different settings of the method argument. But we quickly get better results with
KNN and logistic regression, so it doesn’t make sense to spend too long trying to tune
decision trees for this particular dataset. HOW DECISION TREE MODELS WORK
At this point, we can look at the model and use it to explain how decision tree models work.
> print(tmodel) n= 40518
node), split, n, deviance, yval * denotes terminal node 1) root 40518 2769.3550 0.07379436
2) predVar126< 0.07366888 18188 726.4097 0.04167583 Listing 6.16 Building a better decision tree
4) predVar126< 0.04391312 8804 189.7251 0.02203544 * 5) predVar126>=0.04391312 9384 530.1023 0.06010230 10) predVar189< 0.08449448 8317 410.4571 0.05206204 * 11) predVar189>=0.08449448 1067 114.9166 0.12277410 * 3) predVar126>=0.07366888 22330 2008.9000 0.09995522 6) predVar212< 0.07944508 8386 484.2499 0.06153112 12) predVar73< 0.06813291 4084 167.5012 0.04285015 * 13) predVar73>=0.06813291 4302 313.9705 0.07926546 * 7) predVar212>=0.07944508 13944 1504.8230 0.12306370 14) predVar218< 0.07134103 6728 580.7390 0.09542212 28) predVar126< 0.1015407 3901 271.8426 0.07536529 * 29) predVar126>=0.1015407 2827 305.1617 0.12309870 58) predVar73< 0.07804522 1452 110.0826 0.08264463 * 59) predVar73>=0.07804522 1375 190.1935 0.16581820 * 15) predVar218>=0.07134103 7216 914.1502 0.14883590 30) predVar74< 0.0797246 2579 239.3579 0.10352850 * 31) predVar74>=0.0797246 4637 666.5538 0.17403490 62) predVar189< 0.06775545 1031 102.9486 0.11251210 * 63) predVar189>=0.06775545 3606 558.5871 0.19162510 *
Each row in listing 6.17 that starts with #) is called a node of the decision tree. This decision tree has 15 nodes. Node 1 is always called the root. Each node other than the root node has a parent, and the parent of node k is node floor(k/2). The indenta- tion also indicates how deep in the tree a node is. Each node other than the root is named by what condition must be true to move from the parent to the node. You move from node 1 to node 2 if predVar126 < -0.002810871 (and otherwise you move to node 3, which has the complementary condition). So to score a row of data, we nav- igate from the root of the decision tree by the node conditions until we reach a node with no children, which is called a leaf node. Leaf nodes are marked with stars. The remaining three numbers reported for each node are the number of training items that navigated to the node, the deviance of the set of training items that navigated to the node (a measure of how much uncertainty remains at a given decision tree node), and the fraction of items that were in the positive class at the node (which is the pre- diction for leaf nodes).
We can get a graphical representation of much of this with the commands in the next listing that produce figure 6.2.
par(cex=0.7) plot(tmodel) text(tmodel)