ANEXO SEIS DIRECCION EJECUTIVA DE PRERROGATIVAS Y PARTIDOS POLITICOS

4.2.1

Schemes for Encoding

Representing MDPs with MTBDDs is more complex than DTMCs or CTMCs since the nondeterminism must also be encoded. An MDP is not described by a transition matrix over states, but by a function Steps mapping each state to a set of nondeterministic choices, each of which is a probability distribution over states.

Assuming, however, that the maximum number of nondeterministic choices in any state is m, and letting S denote the set of states of the MDP, we can reinterpret Steps as a function of the form S × {1, . . . , m} × S → [0, 1]. We have already discussed, in

the previous section, ways of encoding a model’s state space S into Boolean variables. If we encode the set {1, . . . , m} in a similar fashion, we can consider Steps as a function mapping Boolean variables to real numbers, and hence represent it as an MTBDD. We will use variables x = (x1, . . . , xn) and y = (y1, . . . , yn) to range over source and destination

states and variables z = (z1, . . . , zk) to encode {1, . . . , m}. We will refer to x and y as row

and column variables, as before, and z as nondeterministic variables. The idea of encoding nondeterministic choice in an MDP as a third index to represent it as an MTBDD was proposed in [Bai98] but the application or practicality of it was not considered.

As discussed in Section 3.3.2, when implementing model checking of MDPs, it is often useful to consider Steps as non-square matrix, where each row corresponds to a single nondeterministic choice. This allows certain parts of the model checking algorithm to be conveniently expressed as matrix-vector multiplication. Similarly, we can think about the MTBDD representation described above as storing a matrix with m · |S| rows, encoded by variables x and z, and |S| columns, encoded by variables y. We will return to this idea when we consider an MTBDD-based implementation of model checking in the next chapter.

First, though, there are still several issues to address with our encoding. For example, it is likely that in some states, there will be less than m nondeterministic choices. Since we represent the MDP by a function of the form S × {1, . . . , m} × S → [0, 1], some parts of it remain undefined. This problem will also arise when we encode {1, . . . , m} into Boolean variables. Unless m happens to be a power of two, we will be introducing extra indices, the meaning of which is undefined. This situation is analogous to the case with DTMCs and CTMCs, where we added dummy states with no transitions to our model. We will take the same approach here, effectively adding extra, empty probability distributions to some (or all) states.

We formalise this idea as follows. Let Steps be an MTBDD over the variable sets x = (x1, . . . , xn), y = (y1, . . . , yn) and z = (z1, . . . , zk). We say that Steps represents an

MDP with state space S and transition function Steps if, for some state space encoding enc : S → IBn, for any s ∈ S:

• if µ ∈ Steps(s), then there exists b ∈ IBk such that:

– fSteps[x = enc(s), y = enc(t), z = b] = µ(t) for all t ∈ S

• for any b ∈ IBk_{, one of the following two conditions holds:}

– fSteps[x = enc(s), y = enc(t), z = b] = 0 for all t ∈ S

– there exists µ ∈ Steps(s) such that

The first part ensures that every nondeterministic choice in each state of the MDP is encoded in the MTBDD and the second ensures that every choice encoded in the MTBDD does actually belong to the MDP. Note that it is acceptable for a choice to be included more than once in the MTBDD. In the definition of an MDP, Steps(s) is a set and hence contains no duplicates. When constructing the MTBDD representing it, though, checking for duplicate choices would be a costly process so we ignore them. Fortunately, it is safe to do this because when we perform model checking on the MDP, we only ever compute the minimum or maximum probability of some event occurring; we never need to distinguish between individual nondeterministic choices.

Another issue to consider is how we actually encode the nondeterministic choices as MTBDD variables. Since the number of nondeterministic choices is typically very small compared to the number of states (i.e. m  |S|), one scheme would be to just enumerate them and encode them as integers, as we would do for an individual PRISM variable. The lesson we learnt from the previous section, however, was that it can be extremely beneficial to capture any high-level structure in our MTBDD encoding.

On example of such regularity is that in nearly all of the MDP case studies we have considered, most of the nondeterminism arises from the parallel composition (scheduling) of modules. Let us assume that this is in fact the only source of nondeterminism. In an MDP composed of m modules, there will be exactly m nondeterministic choices in every state, each corresponding to one of the modules being scheduled. This suggests a more structured encoding, using m nondeterministic variables, one for each module.

We illustrate the difference between these two schemes, which we will refer to as ‘enumerated’ and ‘structured’, through an example. Consider a model comprising four modules. In each state there will be four nondeterministic choices. Using the ‘enumerated’

scheme, we can encode these with 2 Boolean variables since 22 _{= 4. Using the ‘structured’}

scheme, we would require 4 variables. For simplicity, assume that the new variables appear at the start of the overall ordering. Figure 4.2 shows what the two MTBDDs would look like. The grey triangles denote the lower subgraphs of the MTBDDs, the exact structure of which is irrelevant here.

From the diagram, it seems clear that the ‘enumerated’ encoding will always be supe-

rior, in general requiring O(m), as opposed to O(m2_{), nodes to encode the nondetermin-}

ism for m modules. In the next section, however, we will demonstrate that, for variable orderings other than the one used in Figure 4.2, this is not necessarily the case.

(a) z1 z2 (b) z4 z2 z1 z3

Figure 4.2: The two MTBDD encodings for nondeterminism in MDPs: (a) ‘enumerated’ (b) ‘structured’

4.2.2

Heuristics for Variable Ordering

As for DTMCs and CTMCs, it is important to consider the ordering of the MTBDD variables. Since the previous reasoning still applies, it makes sense to retain our ordering heuristics for the row and column variables. All we need to consider, in fact, is where the nondeterministic variables are inserted into this ordering. The choice will depend on which of the two encodings from the previous section we opt for.

The obvious two policies would be to place all the nondeterministic variables either at the very top of the ordering or the very bottom. We will refer to these two options as ‘top’ and ‘bottom’. Of these, ‘top’ would appear most promising option since the nondeterministic variables can be seen as related to all other variables: they affect every transition of the model.

The effect of this option has already been illustrated for each of the two encodings in Figure 4.2. Note that, since the only difference between the two MTBDDs is the encoding of the nondeterminism, the lower regions of the MTBDDs, i.e. all nodes labelled with row and column variables rather than nondeterministic variables, are exactly the same for both encodings. We have already made the observation that the maximum number of nondeterministic choices is typically much less than the number of states in the model. If this is the case, then the lower parts of the MTBDD represent the majority of the graph. Hence, there is actually very little difference between the two encodings.

If we focus on the ‘structured’ encoding, however, we have a third possibility for variable ordering. There are m nondeterministic variables, each corresponding to one module. Recall from the discussion of state space encoding in Section 4.1.1 that the row

Model N States MTBDD Nodes

‘Enumerated’ ‘Structured’

‘Top’ ‘Bottom’ ‘Top’ ‘Bottom’ ‘Middle’ 3 10,744 28,500 28,557 28,503 28,590 23,159 4 201,828 65,621 79,960 65,628 80,061 43,468 Mutual 5 6,769,448 136,836 233,641 136,845 233,777 78,292 exclusion 6 1.3 × 108 _{206,204 506,653 206,219 507,108 103,385} 8 4.5 × 1010 _{381,163 - 381,192 - 153,571} 10 4.7 × 1013 _{962,787 - 962,831 - 328,030} 4 84,096 2,290 1,973 2,297 1,995 1,398 Coin 6 4,612,864 6,948 7,540 6,963 7,605 3,142 protocol 8 2.2 × 108 15,684 25,936 15,713 26,220 5,587 (K = 8) 10 1.0 × 1010 _{29,655 91,990 29,699 92,801 8,718} 12 4.3 × 1011 50,079 342,087 50,145 346,005 12,536 14 1.8 × 1013 _{78,262 1,322,933 78,353 1,339,214 17,047}

Table 4.5: MTBDD sizes for several different MDP variable orderings

and column variables for each module are grouped together. Since the nondeterministic variable for a module can be seen as related to its row and column variables, it would seem wise to group them all together. We will refer to this third ordering option as ‘middle’.

For a system of m modules, where the state space of the ith module is encoded by ni row

variables (xi,1, . . . , xi,ni) and ni column variables (yi,1, . . . , yi,ni), the three orderings can

be summarised as shown below. The parentheses are purely to aid readability.

• ‘top’ (z1, . . . , zm), (x1,1, y1,1, . . . , x1,n1, y1,n1), . . . , (xm,1, ym,1, . . . , xm,nm, ym,nm)

• ‘bottom’ (x1,1, y1,1, . . . , x1,n1, y1,n1), . . . , (xm,1, ym,1, . . . , xm,nm, ym,nm), (z1, . . . , zm)

• ‘middle’ (z1, x1,1, y1,1, . . . , x1,n1, y1,n1), . . . , (zm, xm,1, ym,1, . . . , xm,nm, ym,nm)

We have experimented with combinations of encodings and variable orderings on several case studies which give rise to MDP models. Our findings are typified by the results, given in Table 4.5, for two examples: Rabin’s randomised mutual exclusion algorithm [Rab82] and the coin protocol from Aspnes and Herlihy’s randomised consensus protocol [AH90]. Both are parameterised by N , the number of processes modelled. The latter case study has an additional parameter, K, which we fix here at 8. Further details about these case studies and others we have used can be found in Appendix E. For each MDP, we give its size (number of states) and the number of nodes in the five MTBDDs which can represent it, one for each of the encoding/ordering pairs discussed. A dash indicates that the MTBDD could not be built due to memory constraints.

The results confirm our various predictions. When choosing the ‘top’ or ‘bottom’ ordering, there is very little difference between the two encodings, but ‘top’ consistently

performs much better than ‘bottom’. More importantly, we find that using the ‘structured’ encoding and the ‘middle’ ordering results in the most compact MTBDD. This confirms our most important heuristic: that it is beneficial to exploit structure in the model.

The encoding schemes we have described here only handle the nondeterminism which results from inter-module scheduling. In the general case, there could also be nondeter- ministic choices made locally, within a module. There is usually no particular structure to such nondeterminism, but it is associated with an individual module. Hence, these choices could be encoded in a simple, unstructured fashion, and the nondeterministic variables used placed with the others for that module.

We are aware of two other examples in the literature which consider the representa- tion of MDPs using MTBDDs. Firstly, we mention the work of Hoey et al. [HSAHB99, SAHB00], who implement techniques for decision-theoretic planning. Although the appli- cation domain is very different, the iterative algorithm implemented is similar to ours. In terms of representation, however, due to the nature of their algorithm, it is more conve- nient to store the MDP as a set of several MTBDDs. In our scenario, this would constitute storing separate MTBDDs, one for each of the possible nondeterministic choices.

A second instance can be found in [DJJL01] which presents a technique for model checking a restricted subset of PCTL over MDPs. Although the application domain in this case is very similar to ours, the emphasis of the paper is on the algorithm itself, not the implementation. They state only that the states of the MTBDD and the nondeter- ministic can be encoded into a set of Boolean variables. The compactness of the MTBDD representation is not considered.

4.2.3

Results

We now present some results to illustrate the effectiveness of the symbolic storage schemes for MDPs described in the previous two sections. As for DTMCs and CTMCs, we will consider the amount of storage space required by the MTBDD and compare this to that of the equivalent explicit storage scheme, described in Section 3.6.1. Any time-related issues will be dealt with in the next chapter where we consider model checking. Typically though, operations on smaller MTBDDs will be performed more quickly.

Table 4.6 gives statistics for the two MDP case studies from the previous section: Ra- bin’s randomised mutual exclusion algorithm [Rab82] and the coin protocol of [AH90]. For each MDP, the table gives the size of the model (number of states, number of transitions and number of choices) and the amount of memory required (to the nearest kilobyte) by the MTBDD or sparse matrix to represent it. As before, we assume that each MTBDD node takes 20 bytes. For the sparse storage scheme, the amount of memory required is

Model N States Transitions Choices Memory (KB) MTBDD Sparse 3 10,744 128,934 36,768 452 1,697 4 201,828 3,379,072 912,320 849 43,951 Mutual 5 6,769,448 1.7 × 108 _{3.8 × 10}7 _{1,529 2,094,274} exclusion 6 1.3 × 108 _{3.9 × 10}9 _{8.7 × 10}8 _{2,019 5.0 × 10}7 8 4.5 × 1010 _{1.8 × 10}12 _{4.0 × 10}11 _{2,999 2.3 × 10}10 10 4.7 × 1013 _{2.7 × 10}15 _{5.1 × 10}14 _{6,406 3.4 × 10}13 4 84,096 392,544 336,384 27 6,243 Coin 6 4,612,864 3.2 × 107 2.8 × 107 61 504,239 protocol 8 2.2 × 108 _{2.1 × 10}9 _{1.8 × 10}9 _{109 3.2 × 10}7 (K = 8) 10 1.0 × 1010 1.2 × 1011 1.0 × 1011 170 1.8 × 109 12 4.3 × 1011 _{6.1 × 10}12 _{5.2 × 10}12 _{245 9.3 × 10}10 14 1.8 × 1013 3.0 × 1014 2.5 × 1014 333 4.6 × 1012

Table 4.6: Symbolic versus explicit storage for MDPs computed as described in Section 3.6.1.

We find that MTBDDs can store structured MDP models extremely compactly. In fact, the representation is generally even more compact than for DTMCs and CTMCs. The main reason for this seems to be that there are fewer terminals in the MTBDDs,

implying that there are less distinct values (i.e. probabilities) in the MDPs. This is

not a property specifically of MDPs, but of the type of case studies we have considered: distributed randomised algorithms. Generally the probabilistic behaviour in these systems results from simple random choices, often from coin tosses with values such as 1₂ or 1₄.