In Scenario III we establish a baseline performance using the 625-sample distribution of the KDD99 dataset that has been interpreted into all combinations of the five Threat-Severity levels. Each Local area has submitted their DSP and overall strategy for interpreting the event distribution the 625 samples. The scenario validates the dataset with the training set and then us- es 9-fold cross-validation. Each participant’s accuracy increases as the error decreases using increased learning rates as seen in the single DSP baseline results (Figure 19). A special note highlights that the Baseline metrics are the same as the Single DSP baseline. Notice that the in- creases are slightly different, yet similar. The underlying common threat distribution may ac- count for some of this. The other factor is the local DSP that makes a strategy based on what the event means to them. The local strategy is to win, and right now, the high roller has the accura- cy.
The group baseline dataset achieves a modest 85.00% accuracy rate using a low learning rate of 0.005 (Figure 19). The low learning rates are good to provide the lowest error surface reduction in the hypothesis space where the desired local response matches the ANN’s observed recommended PPL. However, lower learning rates takes longer to train the ANN’s structure. As the learning rate increases for the system, the success rate level out at approximately 96.7%. The error is significantly reduced using a learning rate of 0.3 with this dataset set. There is no signifi- cant change when adjusting the LR to 0.7.
The primary cause of the highest MSE, shown in Figure 19 are related to the DSP con- straints and inconsistent desired PPL responses when faced with tie-breakers found in Area-D’s DSP Majority strategy, where interestingly, order appears to be learned by the ANN to consider and maintain local constraints imposed by Baseline’s DSP. The error went down slightly from a system average of 15.58 to an average of 14 using a LR of 1.0. Area-C’s Local-Only DSP strate- gy has the lowest local error because of the nature of its DSP, which is to recommend the local report only. Using this strategy, Area-C is an active reporter, but only listens to local reports. This strategy may be beneficial in some circumstances for the areas using a similar strategy.
The reporting nature of Area-C still provides the global event from which the global poli- cy is derived. The fact that Area-C is reporting emphasizes the law of large numbers that effec- tively reduces the MSE for the ANN. The Baseline (Area-A) has the second highest rate of error still due to the constraints imposed by the local DSP. Finally, Area-D’s majority DSP strategy remains high in local errors, while maintaining a 94% success rate. A summary of the baseline performance for group DSP’s performance using the four learning rates provided in the next sec- tion. Area-A and Area-B benefit most from a learning rate of 0.7, while Area-C and Area-D show minimal increase in accuracy performance. Participants who choose to only provide threat reports also provide value to the other participants enabling a more accurate picture of a global threat occurrence. Area-C is a reporter only and desires a local reported PPL response only. Ar- ea-C’s reports contribute to the Area-C had a slightly higher performance rating of 96.00% than the error prone Area-D which had the lowest averaging performance rating high of 94.00%.
Figure 19. Group Decision-Support Profile Baseline Results
Figure 24 shows Area-B’s high-roller DSP strategy has the highest PASS rate of 92.10% and the second lowest error of 57.7 using a low ANN learning rate of 0.005. Area-B’s strategy desires the PPL recommendation to best mitigate or avoid the highest occurring global reported threat incident within the global event. Since this desired PPL recommendation is relatively con- sistent despite the order of who is reporting the errors are low. This means that the ANN is ca- pable of determining when to consider threat order. Interestingly, when Area-B has a tie- breaking event, the ANN decides the tie-breaker instead of the local decision-maker as found with Area-D’s majority DSP strategy, which results in a lower localized error. This implies that
Area-D’s DSP strategy that includes a random tie-breaker, it has the highest contribution of system error of 84.4 as shown in Figure 25. Contrasting Area-D’s DSP strategy with The Baseline’s DSP has a constraint that precludes the ANN from providing a “RED” alert unless it is from Area-A, the ANN considers the ordering of “who” is reporting within the training sample events. As a result, area A has the third highest local error among all participants. Area-C and Are-D’s PASS% performance ratings are both approximately 80.20%.
Area-D has the highest error because its DSP strategy incurs a random tie break proce- dure when identical locally determined threat reports meet the same locally defined threat- severity level from the KDD99 threat distribution of report labels. Since the tie break consists one of the threats, and not a standardized choice, this induces localized error which makes the ANN‘s local MSE for the Majority DSP strategy increase from inconsistencies of decisions on the same set of events when order is not considered. It is noted that the ANN is capable of learn- ing Area-D’s strategy type, but needs more training samples to reduce the errors associated with the PPL recommendations. Area-C has the lowest error due to its DSP strategy to only desire PPL recommendation from its local IDPS. In the next section, we assess the generalization accu- racy of the ANN using 9-fold cross-validation on the group-baseline, and testing with fold-9 as the hidden sample dataset.