Quantiles for continuous-time Markov chains can be defined in a similar way as for discrete-time Markov chains (see Section 3.1). However, in the case of CTMCs we
3.7 Reachability quantiles and continuous time
have to consider trajectories rather than paths, in which case the quantile can be a real number (rather than an integer) and min and max in the definitions need to be replaced with inf and sup, respectively. As an example, the quantile
inf t P R : Prs ♦ďtΦ
ě p(
asks for the “smallest’’ time-bound t such that the probability of reaching states statisfying Φ within the given time is at least p starting from some fixed state s.
The theoretical basis for the computation of reachability quantiles in CTMCs relies on an approximation scheme that was already proposed in [Bai+14b]3. The presented scheme consists of a combination of an exponential search with a succeeding binary search in order to find the demanded quantile value. As a very first step, it is checked that the quantile is finite using a precomputation similar to the calculations required for reachability quantiles inDTMCs or MDPs (see Section 3.3.1 or Section 3.4.1). Then,
the mentioned exponential search is used to determine the smallest i P N such that Prs ♦ď2
i
Φ ě p holds for the first time. The existence of such an i can be guaranteed due to the already completed precomputation. If i ě 1, then a binary search is performed in the interval [2i´1, 2i] to find a t P [2i´1, 2i] such that Pr
s ♦ďt´
ε
2Φ ă p
and Prs ♦ďt+
ε
2Φ ě p holds for a user-defined accuracy of ε ą 0. Due to the preceding
precomputation such a t exists, and it corresponds to an ε-approximation of the demanded quantile inf t P R : Prs ♦ďtΦ
ě p(. If it is the case that the exponential search aborts immediately for i = 0, the binary search will be performed in the interval [0, 1].
3.7.1 Dualities
When handling continuous time the only considered models are CTMCs. Since a
CTMC does not involve any nondeterminism, there is no need to distinguish between
existential or universal quantiles.
So, the following dualities can be established when considering increasing state properties: qus Pąp(A U ď? B) =qus Pă1´p( ( A) R ď? ( B ) ) =qus Pě1´p( ( A) Rď? ( B ) ) =qus Pďp(A Uď? B) qus Pąp(A Rě? B) =qus Pă1´p( ( A) Uě? ( B ) ) =qus Pě1´p( ( A) Uě? ( B ) ) =qus Pďp(A Rě? B)
3The group of Monika Heiner planned to integrate an adapted form of this scheme into a development
version of the tool MARCIE [HRS13] in order to support the computation of quantiles in the context of stochastic Petri nets.
For the case of decreasing state properties the following dualities can be introduced: qus Pąp(A Uě? B) =qus Pă1´p( ( A) Rě? ( B ) ) =qus Pě1´p( ( A) Rě? ( B ) ) =qus Pďp(A Uě? B) qus Pąp(A Rď? B) =qus Pă1´p( ( A) Uď? ( B ) ) =qus Pě1´p( ( A) Uď? ( B ) ) =qus Pďp(A Rď? B)
4 Expectation Quantiles
To investigate the interplay of two reward functions (such as one for the consumed energy and one for the achieved utility) we address here path formulas where instead of sets A, B Ď S (as done for reachability quantiles, see Chapter 3), constraints for some other reward function are imposed. For instance, given two reward functions rewe : S ˆ Act Ñ N (for the energy) and rewu : S ˆ Act Ñ N (for the utility), the energy-utility trade-off (as introduced in Section 3.5)
λe,u = ♦ (energy ď e) ^ (utility ě u)
directly relates the accumulation of the reward function for the energy with the accumulation of the reward function for the gained utility.
So, the objective is the minimal or maximal expected value of a random variable f [r] : InfPaths Ñ N Y t8u. For instance, if f[r] is increasing in r and θ stands for some rational threshold, then an expectation quantile can be defined as the least r P N such that the expected value of f[r] is larger than θ for all or some scheduler(s). As an example for quantiles with expectation objectives, we consider a Boolean conditioncond for finite paths and the random variable f[e] = utility|cond : InfPaths Ñ N Y t8u that
returns the utility value that is earned along finite paths where cond holds. Formally: utility|cond(π) =sup rewu pref (π, k) : k P N, pref (π, k) |ù cond(
That is, if π is an infinite path with π |ù ♦cond (i.e., pref (π, k) |ù cond for some k P N) then utility|cond(π) = rewu(ρ), where ρ is the longest prefix of π with ρ |ù cond. If π |ù lcond (i.e., pref (π, k) |ù cond for all k P N) then utility|cond(π) can be finite or
infinite, depending on whether there are infinitely many positions i with rewu(si, αi) ą 0. Given a scheduler S and a state s in M, the expected utility for condition cond is the expected value of the random variable utility|cond under the probability measure
induced by S and s: ExpUtilS
s cond = ÿ
rPN
r ¨PrSs π P InfPaths : utility|cond(π) = r(
Note that ExpUtilS
s cond = 8 is possible if Pr S s ♦l(cond) ą 0. We define ExpUtilmax s cond = sup S ExpUtilS s cond and ExpUtilmin
s cond = infS ExpUtil
S
s cond.
Expectation energy-utility quantiles can be formalised by dealing with conditions cond[e] that are parameterised by some energy value e P N. Examples are the following
quantiles that fix a lower bound u for the extremal expected degree of utility and ask to minimise the required energy:
qus DExpUąu(energy ď?) = min e P N : ExpUtil max
s energy ď e ą u( qus @ExpUąu(energy ď?) = min e P N : ExpUtil
min
s energy ď e ą u(
where a path ρ |ù (energy ď e) if and only if rewe(ρ) ď e. We now want to discuss how to compute expectation quantiles in MDPs with the two reward functions rewe and rewu. The approach will be shown computing
ED
s =qus DExpUąu(energy ď?) and E@
s =qus @ExpUąu(energy ď?)
when the utility-bound u has been fixed. Using known results for standard MDPs, it can be obtained that ExpUtilmax
s (energy ď e) is finite, provided that Prmins (♦(energy ą e)) = 1. If, however, M contains end components where all the state-action pairs have zero energy reward then Prmin
s (♦(energy ą e)) ă 1 and ExpUtilmaxs (energy ď e) = 8 is possible. Therefore, a precomputation is necessary as already used for the computation of reachability quantiles (see Chapter 3).
4.1 Computation scheme
In order to introduce the computation of expectation quantiles we start by restricting ourselves to models where the demanded expectation quantile turns out to be finite and therefore the presented computation will terminate. This renders a precomputation unnecessary for those specific models, and allows to concentrate on the required steps for computing the quantile values.
So, let us make the simplifying assumption that all end components are both energy- and utility-divergent, i.e., whenever (T, A) is an end component of M then there exist state-action pairs (t, α) and (v, β) with t, v P T and α P A(t), β P A(v) such that rewe(t, α) and rewu(v, β)are positive. This assumption yields that Prmins (♦(energy ą e)) = 1 and hence, ExpUtilmaxs (energy ď e) and ExpUtilmins (energy ď e) are finite for all states s P S and arbitrary energy bounds e P N. Moreover, for each scheduler S we have limeÑ8ExpUtilSs(energy ď e) = 8. This yields the finiteness of the expectation quantiles ED
s and Es@. The computation of EsD and Es@ can be carried out using an iterative approach as it was done for reachability quantiles.
For E@
s, we compute iteratively the values us,e = ExpUtilmins (energy ď e) until us,e ą ufor the first time, in which case Es@ = e. It remains to explain how to compute us,e. We can use an LP-based approach and characterise the vector (us,i)(s,i)PS[e] as the unique solution of the LP with variables xs,i for (s, i) P S[e] and the objective to maximise the sum of the xs,i’s (for i ď e) with subject to:
xs,iďrewu(s, α) + ÿ tPS
P (s, α, t) ¨ xt,i´rewe(s,α) if rewe(s, α) ď i for α P Act(s)