Whether the evaluation purpose is for a software component, hardware component, behavioral attribute, or any combination of these, the unavailability of appropriate datasets represents one of the
main challenges to validating the accuracy of models. Unfortunately, the security domain as a study field lacks representative statistical failure datasets [10], [11], [13], metrics [7], [9], [11], [12] whose attributes can be studied and whose quantitative measures can be established and validated, and concrete benchmarks to which model insights can be compared [11]. Sharing accurate, complete failure datasets is very limited due to various concerns, among them, legal liability, competitive advantages or reputation [10]; fear of attackers using such data, or loss of public confidence [13]; or privacy issues [82]. As a result, various assumptions such as the use of hypothetical data and scenarios or statistical distributions are commonly used to build and validate such evaluation models. In a nutshell, validating a security quantification method is an especially difficult issue [7].
In what follows, we briefly demonstrate the common practices used in work addressing modeling for security evaluation. To demonstrate the risk measurement model presented in [11], hypothetical attack graphs and success probabilities are assumed. In [67], Bayesian probability estimates are used to demonstrate a proposed vulnerability analysis of network resources based on attacker behavior. Threshold risk values are also estimated for network resources to define the points where resources are attack prone. The work presented in [61] to perform network vulnerability analysis used simulation techniques to model scenario-specific attack templates, system configuration, and attacker profiles. As characteristics of intruder’s profiles are not available to the work in [3], scenario assumptions are used in the assessment model to exhibit system-level vulnerabilities on a privilege graph, bounding all successful attack scenarios. Also, the work of [13] used a hypothetical enterprise environment and structure to demonstrate the application of attack trees, showing how attacks can be represented in a structured and reusable form. In the security risk methodology proposed in [10], a hypothetical example with data estimates that represent a scenario common in many organizations is used. In the example demonstrated in [78] to model operational risk in financial institutions using Bayesian networks, a hypothetical scenario, including network nodes and their relationships and associated datasets, is assumed.
Some research addresses the dataset generation itself. For instance, the work of [82] presented an approach to dynamically create network intrusion datasets as opposed to one-time-use data. The approach is based on a set of predefined detailed description of intrusion profiles and guidelines that define acceptable datasets. An experimental network setup is used to capture and establish network traffic and intrusion behavior in a testbed environment. Regardless of the quality of such data, the generated traffic and associated anomaly patterns remain restricted to the scale of the experiment.
However, the work of [10] argues that the challenges facing the development of appropriate risk assessment for computing systems are not unique to the IT industry; financial markets and the insurance industry have dealt with risk quantification, irrespective of the uncertainty involved, the unavailability of appropriate statistics, and the technical challenges. However, some, such as [13], argue that attack datasets, although not yet at the preferred level, are becoming more available than before as a result of increased public interest in and media coverage of Internet security. The work of [10] also identified three forces for pushing towards a new security quantitative framework: security insurance needs, avoidance of liability, and market competition. Once insurance claims and compensations start rolling, statistics will proportionally develop, including metrics such as frequency of incidents, losses, and so forth. Furthermore, we argue that with the recent advances in computing paradigms, these forces of change will accelerate significantly. In the cloud paradigm, for instance, the change will initiate especially from the Cloud user’s side with respect to the insurance needs and exposure liability, pushing the establishment for a sensible quantification ground before risk is transferred to the cloud provider.
Today, there are only a few credible statistical reports and surveys about failures, including cyber- attacks, such as those compiled by CERT SEI [83]. In our work, however, failure datasets alone cannot be meaningful for the proposed evaluation methods if not combined with the corresponding system configuration and logical topology with respect to failure. Therefore, to evaluate and validate the proposed work, we have extended the literature review, trying to locate a suitable failure field dataset with system logical diagrams. We have searched available BSTJ Journal, Bell Labs, and NASA historical data; and we have also contacted Google and Cisco with requests for a suitable dataset, but we achieved no success in these endeavors. Moreover, we realize that setting up a representative experimental environment would be very expensive; and would need to run for quite some time in order to establish quality statistics [84], which was not possible either. Nevertheless, major parts of the proposed models follow analytically from reliance on well-founded theories and modelling techniques, such as reliability theory, Multi-State Systems (MSS) and Universal Generating Function (UGF), and Bayesian networks and their inference algorithms.
If more resources and time were available, to establish parameter values of a failure model, one could uses: 1) security auditing tools, as suggested in [7]: vulnerability scanners such as Nessus; network scanners such as nmap; security scanners such as Tiger; and a host scanner such as COPS; and 2) event logging tools: such as Tripwire and InTrust.