Diagnóstico definitivo:

The purpose of this section is to provide a proof for the following proposition. Proposition 5.23. Greedy strategy improvement moves from the strategy σR1B to the strategy σR2.B

For the vertexbi, the fact that greedy strategy improvement does not switch away from the action ai is implied by Proposition 5.22. The following proposition will prove useful throughout this proof.

Proposition 5.24. For every vertex v such that v ₆= bi and v 6= fi, we have ValσR1B (v) = Valσ

B 2i+2(v).

Proof. The only difference between σB

R1 and σ2Bi+2 is that the vertex bi selects the actionai in the strategy σ_R1B . Note that if we select a vertexv such that v6=bi and v₆=fi, and apply the strategyσ_R1B , then the vertexbi will never be reached fromv. Therefore, we must have that ValσBR1(v) = Valσ

B 2i+2(v).

Consider a vertex v in the set V \({bi, fi, x, ci} ∪ {bj : j < i}). Proposi- tion 5.24 implies that we can apply Proposition 5.16 to argue that greedy strategy improvement moves to σB

2i+3 at this vertex. This is because there is no outgoing

edge fromv to bi or fi, and therefore greedy strategy improvement will switch the same actions as it would inσB_2i+2 at the vertex v.

All that remains is to prove that every vertex that has an outgoing action to the vertex fi will switch to that action. The following proposition shows that the vertexfi has a larger valuation than every other vertexfj.

Proposition 5.25. We have ValσBR1(f

i)>Valσ

B R1(f

j) for everyj6=i.

Proof. For every vertex fj where j6=i, Proposition 5.24 implies that Valσ

B R1(f

j) =

Valσ2i+2B (f

j). Therefore, we can apply Proposition 5.19 to argue that, ifk= min(B), then ValσBR1(f

k) >Valσ

B R1(f

j), for every j such that j 6=iand j 6=k. Therefore, to prove this claim it is sufficient to show that ValσBR1(f

i)>Valσ

B R1(f

k).

Ifl= min(B>i∪ {n+ 1}), then we have:

ValσBR1(f

i) = (10n+ 4)(2i−2i−1)−4n−1 + Valσ

B R1(c

Moreover, we can express the valuation offkas: ValσR1B (f k) = X j∈B<i (10n+ 4)(2j−2j−1)−4n−1 + ValσR1B (c l) ≤ i−1 X j=1 (10n+ 4)(2j−2j−1)−4n−1 + ValσBR1(c l) = (10n+ 4)(2i−1₋20)₋4n₋1 + ValσR1B (c l).

Since (10n+ 4)(2i−2i−1) >(10n+ 4)(2i−1 −20) for every i >0 we can conclude that ValσBR1(f

i)>Valσ

B R1(f

k).

Firstly, we will consider the vertex x. Proposition 5.25 implies that x will be switched to the action (x, fi). This is because every outgoing action from x has the form (x, fj), and each of these actions has the same reward. Therefore, the fact that ValσBR1(fi) > ValσBR1(f

j) for all j 6= i implies that Appealσ

B

R1(x, fi) >

AppealσBR1(x, fj) for all j6=i.

Now we will prove the claim for the vertexci. Using the fact that 2i−2i−1>0 for everyigives:

AppealσBR1(c i, fi) = (10n+ 4)(2i−2i−1) + 1 + Valσ B R1(r i) >ValσBR1(r i) = Appealσ B R1(c i, ri).

Therefore (ci, fi) is the most appealing action at the vertexci.

Finally, we will consider the vertices bj with j < i. We will begin by proving that every action at bj other than (bj, fi) is not switchable. Proposi- tion 5.24 implies that, for every action a at the state bj other than (bj, fi), we have AppealσBR1(a) = Appealσ

B

2i+2(a). Since j ∈ B, Proposition 5.16 implies that

greedy strategy improvement would not switch away from the actionaj at the ver- tex bj in the strategy σ2Bi+2. This implies that Appealσ

B

2i+2(a) ≤ ValσB2i+2(b

j), and therefore we have AppealσR1B (a)≤Valσ

B R1(b

that the action (bj, fi) is switchable at the vertexbj.

In the proof of Proposition 5.25 we derived an expression for the valuation of fi in terms of the vertex cl, where l = min(B>i∪ {n+ 1}). We can use this to obtain: AppealσBR1(b j, fi) = (10n+ 4)(2i−2i−1) + Valσ B R1(c l) = (10n+ 4)2i−1+ ValσR1B (c l).

We can also express the valuation of the vertex bj as:

ValσBR1(b j) = (10n+ 4)2j−1 + X k∈B>j_∩B<j (10n+ 4)(2k−2k−1) + ValσBR1(c l) ≤(10n+ 4)2j₋1 + i−1 X k=j+1 (10n+ 4)(2k₋2k−1) + ValσR1B (c l) = (10n+ 4)2j₋1 + (10n+ 4)(2i−1 ₋2j) + ValσBR1(c l) = (10n+ 4)2i−1₋1 + ValσBR1(c l).

Since (10n+ 4)2i−1 >(10n+ 4)2i−1₋1 we have that AppealσBR1(b

j, fi)>Valσ

B R1(b

j).

Since (bj, fi) is the only switchable action at the vertex bj, we have that this action must be switched by greedy strategy improvement at every vertex bj withj < i.