To study how the symbolic synthesis algorithm presented in Section 3.4 performs in practice, we consider the cache coherence protocol called the Valid-Invalid (VI) protocol, which has been described in Chapter 1. This is one of the simplest cache coherence protocols, and is thus conveniently representable as a finite-state protocol. Nonetheless, it is qualitatively representative of the kinds of distributed protocols that we wish to target in this dissertation. We built a prototype tool in OCaml that implemented Algorithm 3.1, using the CUDD [Som15] BDD manipulation library as a back-end.
7Strictly speaking, the algorithm returns a set of valuations over
e
V, which includes variables other than those inGe, but nonetheless, because we haveVe⊇Geby construction, every valuation forVeis also a valuation forGe.
3.5.1
Applying the Symbolic Algorithm to Complete the VI Protocol
We constructed a finite version of the VI cache coherence protocol to evaluate the symbolic synthesis algorithm shown in Algorithm 3.1. In the sequel, we assume that the Directory
fsm-skis of the formD,hLD,lD0,ID,OD,RD,FDs,FDwi, and each Cachefsm-skCiis of the formCi,hLi,li0,Ii,Oi,Ri,Fis,Fiwi. We considered three instances of the completion problem:
• In the first version, the set oftentativetransitions that were added to the directory machine is
restricted to transitions of the formhA, RSP,guard,l0i, for eachl0∈LD, whereguardrepre-
sents a fresh propositional variable to be solved for. Similarly, the set of tentative transitions added to each cache machine was restricted to transitions of the formhB, INV,guard,l0i, for eachl0∈Li, whereguard again represents a fresh propositional variable. In essence, the only synthesis that needs to be performed in this version is to determine the final state l0 that each machine needs to transition to. In this case, symbolic algorithm was able to obtain a correct solution within 30 seconds.
• In the second version, the set of tentative transitions for the directory machine is restricted to transitions of the formhA,m,guard,l0i, for everym∈ID∪OD, and for everyl0∈LD. Similarly, the set of tentative transitions for the cache machine is restricted to transitions of the formhB,m,guard,l0i, for everym∈Ii∪Oiand for everyl0∈Li. This is tantamount to determining whatmessageto send or receive when at the locations labeled A and B, as
well as the next state to transition to after sending or receiving the message. In this case, the symbolic algorithm was able to converge on a correct solution in about ten minutes. • In the third and final version, the set of tentative transitions for the directory machine
includesalltransitions of the formhl,m,guard,l0i, for everym∈ID∪OD, and for every
l,l0∈LD. Similarly, the set of tentative transitions for the cache machine is restricted to transitions of the formhl,m,guard,l0i, for everym∈Ii∪Oiand for everyl,l0 ∈Li. These tentative transitions are added only if they do not result in non-determinism in thefsm-sks. In this version, the completion algorithm does not have any knowledge about the starting locations of the missing transitions are, what messages to send or receive in the starting locations, as well as what the final locations of the transitions are. For this version of the problem, the symbolic algorithm was unable to obtain a correct solution even after six hours of computation time.
To summarize, the three versions of the protocol completion problem for the VI coherence protocol differ in the amount of programmer intuition conveyed to the algorithm. The observa- tion here is that the symbolic algorithm performs better when the search space of solutions is restricted by leveraging the intuitions that a programmer has. We now discuss the reasons for why the algorithm does not scale in the hardest of cases, as well as elaborate on some of the insights obtained from this experiment.
BDDs and the Scalability of the Symbolic Algorithm
In our experiments with Algorithm 3.1, we observed that the BDDs often consisted of tens to hundreds of millions of nodes. The calls to BDD manipulation routines sometimes required tens of minutes of computation. We experimented with enabling the dynamic reordering of BDDs in CUDD. This helped keep the size of the BDDs manageable and enabled quick completion of the BDD manipulation routines. However, the cost of this was that every time the dynamic reordering was triggered, it often took tens of minutes to complete the reordering, based on the internal heuristics implemented in the CUDD library. To summarize, enabling dynamic reordering did not have a positive impact overall execution time of the algorithm. Given that the BDDs were being used to represent constraints over a set containing about 600 variables in the hardest versions of the VI completion problem, we could not exhaustively evaluate all possible static variable orderings. We did however, experiment with a few static variable orderings, that we believed were reasonable, but were unable to improve the execution times of the algorithm.
Impact of Symbolically Retaining all Solutions
Figure 3.1 depicts the reachable state space of the protocol in terms of the interpretation that is chosen (in this case, parameter valuations), at a conceptual level. We have empirically observed that checking if acorrectversion of the VI protocol satisfies all theltlspecifications can be
performed rather efficiently,8and requires only a few seconds of computation time, even with a static BDD variable ordering. From this observation, we infer that the regionGis amenable to being represented compactly using BDDs. However, Algorithm 3.1, first computes the region U, and then computesG, by removing all states (and the parameter valuations which led to their conditional reachability) that can reach an erroneous state in one or more steps. We 8This can be accomplished by simply executing Algorithm 3.1 until (and including) line 13, and checking that
init error
U
G
Figure 3.1: Depiction of the state space of the protocol in terms of all possible completions. The region markedU, which includes all other regions is the state space of the protocol which is reachable if the set of parameter valuations is left unconstrained,i.e., this is the region that
is the union of the reachable state space for every possible completion. The region marked G, which includes the region markedinit, consists of the set of states of the protocol that are reachable if agood completion is chosen. The set U\Gdenotes the set of states that are
reachable if abadcompletion is chosen. These are states that can reach an error state in zero
or more steps, under a givenbadcompletion.
have empirically observed that the BDDs representingGare compact. We thus conclude that representing the large parts of the setUthat are conditionally reachable, together with the parameter valuations which ensure their reachability is difficult using BDDs.
To try and reduce the size of the BDDs representing these intermediate results, our imple- mentation differs slightly from Algorithm 3.1, in the following ways:
• We separateltlspecifications that aresafetyspecifications,i.e., of the formGp, from true
liveness specifications, which could involve eventualities.
• We aggressively eliminate interpretations that are proven unsafe, as early as possible, during the execution of the algorithm. Specifically, the computation in line 3 in Algorithm 3.1 is interleaved with steps to eliminate incorrect interpretations. This is done by eliminating parameter valuations that cause the currently computed under-approximation of the set of reachable states to have a non-empty intersection with the set of states where the invariant is violated.
Unfortunately, this optimization did not have much effect on the execution time of the algorithm, owing to two reasons:
1. We only eliminate a parameter valuation when it has beenprovento reach an unsafe state.
As can be seen from Figure 3.1, there is a large set of states markedU\G, which will inevitably lead to an unsafe state, but might need several steps to do so. This causes our algorithm to retain large parts of the setU\Gas a function of the parameter valuations in a
symbolic form. And we have already discussed that this space is not compactly representable using a static BDD ordering. The problems with enabling dynamic reordering have also been discussed earlier.
2. The aggressive pruning only prunes parameter valuations which violate somesafetyspecifi-
cation. A large part of the specifications for the VI protocol are liveness specifications. We have empirically observed that even after pruning unsafe parameter valuations, the BDDs that evolve during the execution of the loop on line 5 of Algorithm 3.1 are often huge. Based on these observations, we concluded that this symbolic approach, while very elegant, was unlikely to perform well in practice on larger, more complex protocols. We conclude the discussion on this symbolic synthesis strategy by summarizing some key insights which influenced the direction of the research described in the rest of this dissertation.
3.5.2
Insights from Experimenting with the Symbolic Algorithm
• Starting with the set of all possible solutions and paring it down to the set of correct solutions is difficult, especially if the state space of the protocol is maintained symbolically as a function of the current over-approximation of the set of correct solutions. More effective algorithms are possible if we require the algorithms to findonecorrect solution, rather thanallof them,
as we show in subsequent chapters of this dissertation.
• Symmetry in the state space cannot easily be exploited to reduce the size of BDDs. Although there has been work along this direction [CJEF96, EW03, EW05, WBE08], most of these techniques are geared towards checkingctlproperties, and notltlproperties with fine- grained fairness assumptions. The problem is that symbolically representing the orbit
relation between states which are equivalent modulo the symmetry assumptions requires an exponentially sized BDD, which negates any savings obtained by eliminating symmetric states.
• Explicit state model checking techniques seem more promising than symbolic techniques for synthesis. Counterexample Guided Inductive Synthesis [SLRBE05, STB+06, SAT+07, Sol09]
can more readily be applied when using explicit state model checking techniques, as we show in the rest of this dissertation. Further, symmetry in the state space of the protocol can also be more effectively exploited, leading to exponential space savings [ID96, Dil96, ES97, SGE00]. • If a CEGIS technique is used, then a purely depth-first or breadth-first approach during the
Description of the incomplete protocol 1 Buildesm-sks 4 Constraintsψon unknown functions 2 GenerateIsuch thatI |=ψ∧ϕ 5 Instantiate Protocol 6 Check correctness 7 Liveness and safety monitors 3 Correct Protocol Augmentϕwith
constraints from errors
8 Correct? Incorrect? Error traces ϕaugmented with additional constraints
Figure 3.2: Algorithmic scheme of all the solution strategies we discuss. The gray rectangles represent inputs, the blue rounded rectangles represent computation, and the red rhombuses represent decisions. Solid blue arrows represent control and data flow, while the dashed black arrows represent data flow.
the case of the symbolic algorithm, which uses a symbolic, breadth-first search strategy. In the later chapters of this dissertation, we explore heuristics for explicit state model checking algorithms, which lead to quicker convergence of the CEGIS loop to find an interpretation that satisfies the requirements described in Section 2.3.
We conclude this chapter with a brief discussion of the solution strategies described in the rest of this dissertation.