5.5.3.1 AUROC Results
The AUROC scores of the models using RF are as follows:
Original: The treebag model using the original data-set yielded an AUROC result of 0.7045 – hence is slightly the superior model (0.89% greater than the SMOTEd model stated below).
SMOTEd: The treebag model using the SMOTEd data-set yielded an AUROC result of 0.6983.
99
5.5.3.2 ROC Results
As for the ROC graphs, the model’s lines in Figures 5.9 and 5.10, look very similar in terms of area encompassed under the curve, therefore, it is prudent to check the AUROC score to make an empirical determination to as which is the superior model.
The AUROC score of the SMOTEd Model was 0.6983 vis-à-vis the 0.7045 for the Original Model. These AUROC scores are very similar, as they are only about 0.89%
apart. So, despite, the Original Model having an ever so slightly higher AUROC score, both model’s detection accuracies are essentially the same.
Figure 5. 9 ROC of Original Figure 5. 10 ROC of SMOTEd
5.5.4 Random Forests Models
5.5.4.1 AUROC Results
The AUROC scores of the models using SGB are as follows:
Original: The treebag model using the original data-set yielded an AUROC result of 0.6730.
100
SMOTEd: The treebag model using the SMOTEd data-set yielded an AUROC result of 0.7103.
5.5.4.2 ROC Results
As for the ROC graphs, as is evident in Figures 5.11 and 5.12, the model’s line (blue) of the SMOTEd model, runs closer to the Y-axis, thus encompassing a larger area beneath it.
Figure 5. 11 ROC of Original Figure 5. 12 ROC of SMOTEd
5.5.5 Model Comparison
This subsection presents the models’ AUROC, specificity, and sensitivity in a tabulated fashion. The tables presented below allow for convenient comparisons to be made in order to deduce whether using SMOTE yielded empirically superior models vis-à-vis models created using the original data-set.
101
Table 5.4 below shows the simple averages of the sensitivity and specificity scores for all of the models created. As is evident, all the models using the SMOTEd data-set have outperformed the models using the original data-set.
Table 5. 4 Models’ Specificity & Sensitivity Average Result Comparison Data/Model DT Treebag RF SGB
Original Data 57.15% 57.36% 63.81% 50.00%
SMOTEd Data 61.54% 63.88% 64.10% 63.80%
Table 5.5 below shows all of the models’ AUROC scores. As is evident, the models using the SMOTEd data-set have yielded a higher AUROC score for all the data-sets, except for the RF model, which is only 0.89% greater, thus essentially the same score.
Again, this empirically proves SMOTE’s superiority vis-à-vis the original data-set.
Table 5. 5 Models AUROC Result Comparison Data/Model DT Treebag RF SGB Original Data 0.5794 0.5736 0.7045 0.6730 SMOTEd Data 0.6179 0.6388 0.6983 0.7103
These results clearly indicate that building models using the SMOTEd data-set yields empirically superior results to those using real data. This is showcased in two areas, firstly, both the AUROC, and the specificity and sensitivity averages, yielded higher scores for the models using the SMOTEd data-set (except for the RF model, as they are almost the same); and secondly, the increase in accuracy across the various models conform more so with the literature when using the SMOTEd data-set as opposed to real data. This is in terms of the predictive accuracy of tree ensembles over single tree techniques (RFs/SGB>treebag>DTs). As is clear in Table 5.5, all tree ensemble models using the SMOTEd data-set outperformed the models using the original data-set, as measured by the AUROC criterion. The results also show that even with the recursive partitioning models’ resilience to class imbalance, using a SMOTEd data-set yields more accurate detection accuracy scores. This is an
102
important finding that contributes towards the literature through recommending the use of SMOTE even when using machine learning techniques due to empirically superior results, as was shown in this chapter.
5.6 Conclusion
This chapter presented the application of Synthetic Minority Oversampling Technique (SMOTE) to an imbalanced data-set comprising 748 Australian mining – 631 of which are financially healthy and 117 are distressed. Four machine learning tree-based techniques were used to create the models for this study. For comparison purposes, the models were trained on two data-sets, the original imbalanced data-set and a balanced SMOTEd data-set, in order to empirically deduce the detection accuracy of SMOTE. A holdout sample using real-life data were used to test the accuracy of the aforementioned trained models using both data-sets. The results indicated that despite the SMOTEd data-set being around 80% smaller than the original, it resulted in superior detection accuracy. This was measured by AUROC, specificity, and sensitivity results. The AUROC results showed the superiority of SMOTE for the DT and SGB models, as for RF, the scores were almost identical pre and post SMOTE.
This study has showcased that using SMOTE is not only easier to handle due to the smaller set, but is also empirically superior to the original class imbalanced data-set. This research has contributed towards the literature by investigating the detection accuracy of SMOTE using a multi-approach system and recommending the use of SMOTE even when using machine learning techniques due to empirically superior results. This chapter has verified Hypothesis 4.
103