There are several approaches to regression analysis. While some techniques yield similar results, others may result in different meta-models. Two techniques are used in this analysis: logistic regression and least squares regression. Additionally, partition trees are used to compare the significant factors with the significant factors of logistic regression and least squares regression meta-models.
a. Logistic Regression
The only MOE in this analysis is mission success. Therefore, the result of each scenario run is either a success or a failure. This produces a binary result, and fitting a logistic regression is a proper way to deal with binary response. The distribution of the response in the raw data is shown in Figure 15. Among 102400 replications, friendly assets achieved mission success in 74066 of the runs. The probability of the success in overall replications is around 0.72.
Figure 15. Distribution of the response.
The summary of the logistic regression model with all the main effects is shown in Figure 16. The p-value of this model is less than 0.0001, which proves that the saturated logistic regression model is significant when compared to the null model.
Figure 16. Summary of the logistic regression model.
The leading 15 factors of the logistic regression model sorted in the order of significance from high to low are shown in Figure 17. The first five factors in the list seem to have a huge impact on the response surface, whereas the other factors have much less impact. Moreover, whether or not the OPV main gun is enabled appears to be the most significant factor of all.
Figure 17. Parameter estimates for the logistic regression.
The purpose of this analysis is not predicting the mission success of an OPV in an ASUW mission, but rather understanding the factors affecting the mission success. Nevertheless, it is still possible to assess how well the meta-model predicts the mission success. The Receiver Operator Characteristic (ROC) curve is one way to evaluate the meta-models. As a rule of thumb, when the area under the ROC curve is between 0.8 and 0.9, the meta-model is considered to be predicting well; when it is higher than 0.9, the meta-model is considered to be predicting excellently (Hemmingsson, Uddén, & Neovius, 2009). The ROC curve of the logistic regression model is demonstrated in Figure 18. The area under the curve is 0.89, which proves that the meta-model is predicting fairly well.
Figure 18. The ROC curve for logistic regression model evaluation.
b. Least Squares Regression
For any specific design point, even though all the values of the input parameters are the same, each simulation run (called a replication) produces different results due to the randomness embedded in the modeling platform. A designed experiment gives the analyst an opportunity to investigate a range of outcomes based on different parameter settings, but it is sometimes better to simplify the data to gain more insights. For instance, when dealing with a binary response, it is possible to take the mean of the response for all the replications at each design point. Then the aggregated average response values can be used to predict mission success. Taking the mean of the response for all replications of each design point yields a multi-level response ranging from zero to one. In this way, the analyst can fit a regular regression model to the aggregated data, and this might provide a better insight into the problem at hand because it is easy to interpret.
The distribution of the mean of the response in the aggregated data is shown in Figure 19. The overall proportion of mission success is 0.7233, which is equal to the proportion of mission success in the raw data. It is also worth
noting that there is a large number of instances where the proportion of mission success is greater than 0.95 or lower than 0.05. This means it will be more difficult for a polynomial model to do an effective job of fitting the aggregated response data. Even so, it is interesting to see what factors are most important.
Stepwise regression methods and expert judgment are used to select the significant factors to fit the least squares regression model. The threshold for the p-values, or the significance level, which is used as evidence of statistical significance for terms included in the models (Htoon & Chan, 2010), is set to be very low to avoid over-fitting. This low p-value threshold also reduces the chances of “false positives” (Type I errors) when identifying important factors and allows us to have interpretable results. It is also true that for large data sets, such as those resulting from large-scale simulation experiments, regression models can contain terms that are statistically significant but not practically important. A low p-value threshold is useful for obtaining relatively parsimonious models in these situations.
Figure 19. Distribution of the mean response. The total number of occurrences (N) is 1024. Among those occurrences, 187 of them resulted in with
(1) Main Effects Model. The Actual by Predicted plot and the Summary of the stepwise model, fitted with only the main factors without any interactions, are shown in Figure 20. Only 15 factors, which proved to be significant, are included in the model. The R-Square value is around 0.56. This value represents how much variability can be explained using this model. There is considerable randomness, as is often the case in the combat models, but the regression terms can help the analyst focus on what really affects the mission success.
Figure 20. Actual by predicted plot and the summary of the fit for the main effects regression model.
The sorted parameter estimates of the main effects model are shown in Figure 21. Note that the first five factors have a more significant effect on the probability of mission success when compared to others, as can be seen by the magnitudes of the t ratios. These five factors are exactly the same factors that showed up in the logistic regression model, as expected.
Figure 21. Sorted parameter estimates of the main effects regression model.
(2) Second Order Model. The Actual by Predicted plot and the Summary of Fit built with the main effects, two-way interactions, and polynomials are shown in Figure 22. The R-Squared value of the second order model is higher than that of the main effects model, even though the second order model contains one less term and involves fewer factors. One might think that is why the R-squared value is not very high.
Figure 22. Sorted parameter estimates for the stepwise regression model to include the main effects, two-way interactions, and polynomials.
The sorted parameter estimates are shown in Figure 23. The six most significant factors are the same as the factors of the main effects model, with a slightly different order.
Figure 23. Sorted parameter estimates of the second order model.
Some terms of the second order model are polynomials and two-way interactions. For example, range to start counter measures is a quadratic term. That is, keeping all other factors constant, changing the value of this term changes the value of the mission success non-linearly.
When doing a regression analysis, it is also possible to fit the model with more degrees of interactions and polynomials. As a part of this regression analysis, three-way interactions were tested as well; however, they did not prove to be significant.