DESARROLLO DE ACTIVIDADES

As discussed previously, we have been careful to—where possible—align SAFE with STPA so that the two can be used together on different parts of the same system. When, though, should an analyst use SAFE instead of STPA? This is to some degree a judgement for the

Pulse  Oximeter   PCA  Pump  

App  Logic  

Pa5ent   Clinician  

Clinician  Instruc-ons   Clinician  Feedback   Health  Outcomes  

PCA  Interlock  Scenario  in  Opera5on   Hospital  Clinical  Opera5ons  

Ite ra5 on   and   Con5 nue d   Devel op men t   Procedure  Revisions   So<ware  Updates   Hardware  Updates   Clinician  Feedback   Health  Outcomes  

App  Development   Hospital  Opera5ons  

Congress  and  Legislature   Regulatory  Authori-es   Congress  and  Legislature  

(e.g.,  FDA)   Regulatory  Authori-es   (e.g.,  UL)   Vendor  Management   Development  Team   App  Design   App  Implementa-on     and  Assurance  

Hospital  Business  Opera-ons   Hospital  Clinical  Opera-ons  

Clinical  Assump-ons   Clinical  Procedures  

App  In  Context  

Figure 4.4: The PCA interlock loop in the clinical context, adapted from figure 4.4 of Leveson [30]. SAFE would apply only to the shaded region.

analyst herself to make, though we believe SAFE should be used when both a) a systems theoretic model of causality begins to hinder analysis more than it helps, and b) components can accurately decomposed/refined into a collection of cooperating automata. A view of the PCA Interlock scenario in the full clinical context is shown in Figure 4.4; we imagine that SAFE would be most useful in the shaded region. For absolute clarity, this section discusses research in the areas of causality and decomposition, but we believe that the direct use of this research will be necessary only in very rare circumstances.

Causality

Leveson writes that “The definition of accident causation needs to be expanded beyond failure events so that it includes component interaction accidents and indirect or systemic causal mechanisms.” [30] We agree that both component interaction and indirect causes are important to consider, but STPA’s literature provides no clear definition of causality. In STPA’s hierarchical control structures (like Figure 4.4), what do the arrows between components mean, precisely? Do arrows between two software- or hardware-based elements

signify something different than those between social constructs like organizations?

At all levels of a system’s sociotechnical hierarchy, arrows between components intuitively read as communication, or more precisely a causal relationship between observable actions of the sender and the behavior of the receiver. Can we be more precise about the semantics? Similarly, what is meant by the lack of arrows between two elements? It would be incorrect to state that the app logic does not affect the patient, but there is no arrow connecting the two. We believe that the semantics of arrows between software- and hardware-based elements of a system’s control structure diagrams should be those of actual, intransitive causality.

Actual Causality We agree with Leitner-Fischer, who writes that actual cause, as pro- posed by Halpern and Pearl, is the correct model of causality for reasoning about safety- critical systems [89, 83]. Leitner-Fischer explains that actual cause theory is a refinement of, and addresses problems with, the popular counterfactual (or alternative world ) style of reasoning, which he explains as:

The “naive” counterfactual causality criterion according to Lewis is as follows: event A is causal for the occurrence of event B if and only if, were A not to happen, B would not occur. The testing of this condition hinges upon the availability of alternative worlds. A causality can be inferred if there is a world in which A and B occur, whereas in an alternative world neither A nor B occurs.

Unfortunately, Leitner-Fischer explains that there are a number of problems with this naive approach, including: a) the failure to identify conjunctions or disjunctions of events as causal, b) preemption of one event by another (and event ordering in general, a major focus of [89]), as well as c) event non-occurrence and irrelevance4_{. In order to address these} problems, the counterfactual model was extended by Halpern and Pearl to what they term

4_{We note that these are similar to many of the problems with existing hazard analyses that drove Leveson}

the structural equation model (SEM)5_{. The SEM presents situations as “logical combinations} of events as well as a distinction [between] relevant and irrelevant causes.” [89]

Halpern and Pearl specify a formalized, three-part test to determine if some event is an actual cause of another event. This test is quite detailed, and not fully germane to this work, so we do not reproduce it here. We recommend, however, the interested reader to both Halpern and Pearl and Leitner-Fischer’s works for a full description and formalization of actual causality [83,89].

Intransitivity At first, using the alternate-world semantics of actual causality seems to lead to a strongly-connected control structure, since each element in the control structure clearly affects each other element. That is, in an alternate world where there is no app logic, the patient may experience a PCA overdose, so why is there no arrow directly connecting the app and the patient? Indeed, every square in the shaded region Figure 4.4 would have to have an arrow pointing to every other square, which is unsatisfactory and confusing.

We believe a reasonable solution to this problem is intransitive noninterference, which comes from the information flow community [91]. Rushby explains that “The idea of non- interference is really rather simple: a security domain u is noninterfering with domain v if no action performed by u can influence subsequent outputs seen by v.” [92]. Rephrased into the domain of elements, communication, and observability, then, we might say that “an element e is noninterfering with another element f if no action performed by e can influence subsequent outputs seen by f .”

We believe that the existence of an arrow in a system’s control structure should signify intransitive observability. If one element is reachable from another (i.e., some path exists between them) that would signify (exclusively) transitive observability. That is, interaction between two components that are not directly connected would necessarily be mediated by all intermediary components. This aligns with our intuition: the only information the pump

5_{We note that the SEM bears some resemblance to the formulae created by Thomas in Section 4.3 of}

receives from the sensors in Figure 4.4 would be via commands generated by the app based on the sensors output. This notion of intransitivity of interference/observability becomes especially important when we consider the propagation of errors between components, a topic addressed in some depth in Section 5.5. We note that the arrows in STPA’s diagrams model interactions—primarily information or command flow, but also occasionally physical phenomena. Causality can often be derived from these interactions.

Decomposition and Refinement

The entire PCA Interlock scenario will likely be thought of as one element by the hospital management, and it is possible that another implementation of the same scenario could satisfy the same goals. That is, a number of systems could suffice for the objectives of the shaded region of Figure 4.4. They would share the same inputs and outputs but could be composed of different devices and application logic. Indeed, this interchangeability is at the core of the component-based MAP vision. How, though, can we know the refinement from one element to a set of subelements is sound? We aim to answer this question with the firm notion of decomposition and refinement from Shankar’s Lazy Compositional Verification [84].

The hierarchical safety structures used in both STPA and SAFE (i.e., Figure4.4) can be thought of as models of decomposition and refinement. Decomposition is a term for breaking a system down into its component parts, while refinement is a related term that signifies increasing the level of specificity of a model into a more detailed specification, sometimes down to the implementation level. A good example of decomposition is moving down an abstraction hierarchy, i.e., in the language used to describe MAP apps from Section 3.3, from the AADL System level to the AADL Process level. Similarly, the code gener- ation aspects of the MDCF Architect (from Section 3.4) is a good example of refining a component’s architecture down to the implementation level.

Sensor:     Pulse  Oximeter   Actuator:   PCA  Pump   Controlled  Process:   Pa7ent   Auto.  Controller:   App  Logic   Pump-­‐>Pa7ent:   IV  Line   App-­‐>Pump:  

Tickets  to  Pump   Sensor-­‐>App:    SpO2  to  App  

Connec7on   Component   Link   Element   Key   Pa7ent-­‐>Sensor:   Infrared  light  to  PulseOx  

App  Logic’s  Successor   App  Logic’s  Predecessor  

Figure 4.5: An expanded view of the shaded region from Figure 4.4, showing components, connections, and the links between the two

we believe that similarly rigorous notions should be used in SAFE as well. Using such formalisms, we could know if the shaded region in Figure 4.4 could be safely replaced by a different process that claims to achieve the same goals, i.e., could uphold the same invariants. This is a topic we explore in some depth in Section 5.4.