When one considers a system on the scale of the vulnerability discovery and disclosure system an appropriate and robust method that provides an understanding of that system is critical. System Dynamics deals with the simulation and interaction between objects in dynamic systems (Forrester, 1958). System Dynamics also provides a robust platform to bring together both qualitative and quantitative data that has been collected, utilising techniques such as causal loop diagramming, stock and flow models and time series simulations. System Dynamics also provides an analytical framework to analyse elements of a system that are organised for a purpose. and emphasises the actions of information, action and feedback (Coyle, 1996). The notion of
feedback loops is crucial within System Dynamics, and takes two forms: positive and negative. Coyle describes both loops as:
“A positive feedback process, or loop, is one which acts to reinforce a change in a system level….a negative loop is goal seeking, that is it tries to move a level toward a desired target”.
(Coyle, 1977, pp.38–40) The System Dynamic approach also allows the investigation of how different variables affect how the system behaves over time. It allows the creation of a dynamic model as the artefact of interest, with the value of the artefact demonstrated via the execution of the model under differing conditions.
3.4.4.1 Model Construction and Simulation
The final phase within the mixed methods framework is to build upon the analysis undertaken in the previous steps and construct a conceptual model of the vulnerability discovery system. This is a critical step as it allows the visualisation of the system and identify key elements and information flows around the system. The initial representation of the system is via Causal loop diagrams (CLDs) which are based upon the thematic analysis, highlighting themes, subthemes and feedback loops that are present within the system. In the case of the vulnerability discovery and disclosure system there are several hypothesised interrelated system archetypes, information flows, causal links that come together to form a larger model.
Using the CLD technique to visualise structures and identify loops that are derived from analysis is a key aspect of this investigation. The interaction of two or more balancing and reinforcing loops is suggested to cause identified behaviours within systems. These archetypes may cause oscillations, exponential growth or ‘s-shaped’ growth within a system.
System Dynamic modelling has been used to understand real-world systems and problems since its creation in the 1960’s (Forrester, 1975). As part of the evolution of System Dynamics, many complex systems have been shown to
produce similar behaviours although the systems under scrutiny are very different; and are known to confirm archetypical dynamic hypotheses and explain generic system behaviours (BenDor and Kaza, 2012; Senge, 1990, pp.379–390)
In the context of the VDDS there are a previously observed influences and interactions within the system that are hypothesised to makeup a diverse set of loops and archetypes. However, the main behaviour that has been shown to be exhibited by the system is the S-shaped growth curve (Alhazmi et al., 2005). This S-shaped curve, known as logistic, suggest that the underlying process that causes this growth to occur is a combination of both balancing and reinforcing loops which initially drive growth, then reduces growth when a limit has been reached. Using simple archetypes is a possible way to capture the underpinnings of seemingly impenetrable levels of complexity, and the interactions of the participants who exist within the VDDS. As stated the VDDS is hypothesised to be made up of entities, interactions and delays. However, to move further and begin to understand the systemic behaviours how the VDDS operates, and crucially how variables impact each other, we must construct a model of the VDDS.
3.4.4.2 The VDDS as a Dynamic System
Within the final phase of this research and to accurately model the VDDS we must first adopt a specific nomenclature to describe the aspects of the VDDS correctly. To this end the terminology used by Voinov (2008) has been adopted when entities, relationships and interactions are described. Initially we start with describing an element which is considered to be a building block of a system, and is considered to have both properties and features (Voinov, 2008, p.7). Features are a distinctive property of system (Peter interacts with Paul), whereas a property is an attribute (ie colour) of a feature/ Interaction is defined again by Voinov by describing the type of relationship that may exist between elements; specifically flows of material and information (Voinov, 2008, p.9). These definitions of interaction are key when modelling the VDDS as we are essentially looking to understand the rules of the system as a whole (Voinov,
2008, p.7). The representation of flows that are present within a system, combined with the elements are the important essence of the VDDS as they represent the disclosure choices that are made, and rewards that are given. In common with other systems that exist, several interactions have been identified within the VDDS, and these sets of processes clearly operate to facilitate the discovery and disclosure of software vulnerabilities.
Adding to Voinovs’ (2008) definitions we can further characterise potential processes by using system dynamic modelling vocabulary defining two further aspects of a system, state and resource. Resources within System Dynamics are defined as is anything that has value which can be transferred from one element to another (Sterman, 2000, p.127). Alongside resource, state of a particular resource in the context of system dynamics can be defined as “any accumulation of that resource” (Wolstenholme, 1996, p.12). By applying and characterising a system in these terms it is a small, yet important leap to introduce a further term, stocks. Stocks within System Dynamic models are measurable amounts of a resource at any given, state and time (Wolstenholme, 1996, p.12). Applying a systems approach, nomenclature and vocabulary we can define in a broad sense the VDDS as:
‘The VDDS converts raw vulnerabilities (State1: Undiscovered
Vulnerabilities) into refined vulnerabilities (State2: Discovered
Vulnerabilities) to be either sold for profit (State3: Money) or disclosed for free (State4: Knowledge)’.
Figure 6 – State Transition of Deriving Value from Vulnerability
A key aspect of the transition between states are the rate in which the conversion between states occur (Wolstenholme, 1996, p.12). The rate at which the conversion occurs are represented within the system dynamic modelling
process as rate variables and are key to understanding the behaviour of the system. In particular, we can see that vulnerabilities transition between states linearly.
3.4.4.3 Why Systems Dynamics?
The construction theory and model that is intended to represent real-world phenomena must start at first principles, or as close as realistically possible if current theory is not adequate. It is also reasonable to assume that the any model will be continuously improved and future generations will represent reality more accurately. Given these assumptions, many modelling approaches exist to assist and suit differing classes of problem when trying to understand the behaviour of a system (Kelly et al., 2013). System Dynamics provides a robust approach for bringing together both the collected qualitative and quantitative VDDS data by utilising techniques such as causal loop diagramming, stock and flow models and time series simulations. System Dynamics deals with the simulation and interaction between elements within in a dynamic system and crucially integrates the concept of feedback whereby elements may influence each other, in positive and negative ways. (Coyle, 1996, p.2; Forrester, 1958). Coyle (1977) describes this influence in terms of processes or loops:
“A positive feedback process is one which acts to reinforce a change in a system level….a negative loop is goal seeking, that is it tries to move a level toward a desired target” (Coyle, 1977, pp.38–40).
The System Dynamic concepts outlined by Coyle (1977), Wolstenholme (1996), Sterman (2000) and Forrester (1975) for the investigation of how different structure, resources and rates affect how the system behaves over time provides a very powerful investigative framework. The framework provides tools for the creation of dynamic models, characterisation of the artefact being investigated with the behaviour of the artefact demonstrated via the execution of models under differing conditions (Sterman et al., 1997). System Dynamics also provides an analytical framework to assess elements of a system organised for a purpose and emphasises the information, action, feedback paradigm (Coyle, 1996, p.5). This concept of feedback is crucial when we consider the systemic
nature of the VDDS, as it these types of flows that ultimately govern how the system behaves and how it will continue to do so in the future.
3.4.4.4 The System Dynamic Modelling Process
To model any system or process the analysts must adopt a process as to how to go about investigating the phenomena (Sterman, 2000, p.85). Consequently, several process steps have been put forward by authors outlining general steps that are followed in constructing a System Dynamic model. The most comprehensive is outlined by Sterman (2000), which brings together processes steps from both Coyle (1996, p. 11) and Wolstenholme (1996, p. 26-19):
Step 1: Problem Articulation (Theme Selection; Key Variables; Time Horizon; Problem Definition)
Step 2: Formulation of Dynamic Hypothesis (Initial Hypothesis Generation; Endogenous Focus; Mapping)
Step 3: Formulation of a Simulation Model (Specification; Estimation; Tests)
Step 4: Testing (Comparison to Reference Modes; Robustness Under Extreme Conditions; Sensitivity)
Step 5: Policy Design and Evaluation (Scenario Specification; Policy Design; What if...; interaction of Policies)
(Sterman, 2000, p.86) As stated by Senge (1990) there are six modes of behaviour that have been identified within the System Dynamics movement. The archetypes are characterised by feedback loops and interactions of those loops in one or more ways. Furthermore, modes of behaviour have been characterised, and by using influence diagramming and codified so that they can be readily identified (BenDor et al., 2012; Sterman, 2000, pp.264–291). Of the fundamental systemic archetypes that exist all are grouped around the growth and decline of a value or an observed variable, for example the growth of bacteria in a petri dish (Goodwin, 1970). These dynamic system archetypes are postulated to
exist within the VDDS, taking the form of loops of information or movement of resources within the system. The commonly found archetypes, along with structures that are found within systems are outlined in Table 6 below.
System Archetype Governing Equation Diagram 1. Linear Growth 12 1#= 3 2. Exponential Growth 12 1#= 45 3. Goal Seeking Growth 12 1#= & − 2 3 4. Logistic Growth 12 1#= 32 1 − 2 & 5. Oscillations 12 1#= 35 12 1#= 372
6. Overshoot and Collapse 12 1#= 382 − 392 : &; 12 1#= 4;2
Table 6 - Systemic Archetypes Adapted from (BenDor et al., 2012; Senge, 1990) Note: S = stock, C= goal/carrying
capacity, k = constant, Ks/Kr = S or R related constants.