Elaboración de la documentación del Sistema de Gestión de la Calidad

Let X be a Markov chain with state space E and transition matrix P. Let R(i, j) be the expected number of visits to state j starting at i, and let F(i, j) be the probability of ever reaching j starting from i; these were defined by (5.2.12) and (5.2.6). We first take up the computation of the potential matrix R.

Suppose j is a recurrent state. Then by Definition (5.3.1), F(j, j) = 1; and by Corollary (5.2.13) this implies that R(j, j) = + ∞. Furthermore, by the same corollary, if j can be reached from i, then F(i, j)<

0 and R(i, j) = ∞ again. If, on the other hand, j cannot be reached from i, then F(i, j) = 0 and R(i, j) = 0. Hence, for j recurrent,

If j is transient and i recurrent, then by Theorem (5.3.12) j cannot be reached from i; therefore, in that case F(i, j) = 0 and

Thus, the only remaining case which is not trivial is when both i and j are transient.

Let D denote the set of all transient states, and let Q and S be the matrices obtained from P and R, respectively, by deleting all the rows and columns corresponding to the recurrent states; that is,

If the states are labeled so that the recurrent states precede the transient ones, then as in (5.3.15), we can write P in the form

by defining K and L suitably. Then for any m ∈

where Q^m is the mth power of Q (similarly for K^m but not for L_m). Thus, from (5.2.15),

which shows that

Now the computation of S is not difficult: noting that

we have

that is, S satisfies the two systems of linear equations (1.7), and we may use either one to solve for S.

In particular, if D is finite, (1.7) shows that the finite-dimensional matrix 5 is the inverse of the matrix I − Q; that is,

(1.8) PROPOSITION. If there are only finitely many transient states, then

On the other hand, when the set D of transient states is infinite, it is possible to have more than one

solution to the system (1.7). In that case, the next theorem shows that S is the smallest possible solution:

(1.9) THEOREM. S is the minimal solution of

Proof. We know by (1.7) that S satisfies (1.10). To show that it is the minimal solution, let Y be another solution. Then we have

Replacing Y on the right by I + Q Y, we get

and repeating thus,

for any , since Q^{n + 1} ≥ 0 and Y ≥ 0. Now taking limits in (1.11) as n → ∞, we get Y ≥ S as claimed.

If Y is another solution, then Y = I + QY and S = I + QS together imply that H = Y − S ≥ 0 satisfies H

= QH. Then every column of H satisfies h = Qh, h ≥ 0. Moreover, every column of S is bounded (since S(i, j) = F(i, j)S(j, j) for i ≠ j). It follows that S is the unique solution of (1.10) if and only if the only bounded solution of h = Qh is h = 0, or equivalently, if and only if

implies h = 0. For further discussion of this point we refer the reader to Section 4 (see also Section 2 of Chapter 7).

(1.12) E^XAMPLE. Let X be a Markov chain with state space E = {1, 2, 3, 4, 5, 6, 7, 8} and transition matrix

States 1, 2, and 3 are recurrent non-null aperiodic and form an irreducible class. States 4 and 5 form another irreducible class of recurrent non-null aperiodic states. States 6, 7, and 8 are transient; from these states only 1, 2, and 3 can be reached.

Here

so using Proposition (1.8) above, we get

Thus, the potential matrix R for this Markov chain is

Next we consider the computation of the probability F(i, j) of ever reaching j from i. If i and j are both recurrent and belong to the same irreducible closed set, then by Lemma (5.3.11),

If i is recurrent and j transient, or if i and j are recurrent but belong to different irreducible sets, then

If i and j are both transient, then R(i, j) < ∞, and by Corollary (5.2.13) we have

for i ≠ j, and the R(i, j) may be obtained by using (1.8) or (1.9) above.

The only remaining case is when i is transient and j recurrent. The computations are simplified somewhat by the following.

(1.16) LEMMA. Let C be an irreducible closed set of recurrent states. Then for any transient state i,

for all j, k ∈ C.

Proof. For j, k ∈ C, by Lemma (5.3.11), F(j, k) = F(k, j) = 1. Thus, once the chain reaches any one of the states of C, it also visits all the other states. Hence, F(i, j) = F(i, k) is the probability of ever entering the set C from i.

In view of this lemma we need only talk about the probability of reaching an irreducible recurrent set. Let C₁, C₂, . . . be the irreducible recurrent classes, and let D be the set of all transient states.

Suppose the states are labeled so that the states in C₁ precede those in C₂, and so on, and the transient states are after all the recurrent ones. Then the transition matrix P is in the form (5.3.15). Since we are interested only in ever reaching C_j, we may lump all the states of C_j together to make one absorbing state. The matrix of transition probabilities then becomes

where

The probability of ever reaching the absorbing state j from the transient state i by the chain with the transition matrix is the same as that of ever reaching C_j from the transient state i by the original chain with the transition matrix P.

We now rewrite as

by defining B as the matrix with columns b₁, b₂, . . .; that is,

Then, for any the nth power of is

where

Evidently, B_n(i, j) is the probability that starting from i, the chain enters the recurrent class C_j at or

before the nth step. Hence, if we put

then G(i, j) is the probability of ever reaching the set C_j from the transient state i; that is, we have proved the following

(1.22) P^ROPOSITION. Let Q be the matrix obtained from P by deleting all the rows and columns corresponding to the recurrent states, and let B be defined as in (1.19) for each transient i and recurrent class C_j. Compute S by using either (1.8) or (1.9), and put

Then for each transient state i and recurrent class C_j,

for all k ∈ C_j.

If there is only one recurrent class and if there are only finitely many transient states, then the computations involved disappear. For in that case, the matrix B becomes a column vector, and since P1 = 1, we have B + Q1 = 1, or B = 1 − Q1. Putting this in (1.20), we obtain B_n = 1 − Qⁿ1, so in the limit we have G = 1 − lim_n Qⁿ1. Since ∑ Qⁿ < ∞, lim Qⁿ = 0, and since there are only finitely many states, Q is a finite-dimensional matrix. Thus lim_n (Qⁿ1) = (lim Qⁿ)1 = 0, and G = 1. Hence we have also proved the following.

(1.23) C^OROLLARY. Let i be a transient state from which only finitely many transient states can be reached. Furthermore, suppose that there is only one recurrent class C which can be reached from i.

Then F(i, j) = 1 for all j ∈ C.

(1.24) E^XAMPLE. In Example (1.12) above, from the set of transient states {6, 7, 8} only the recurrent irreducible set {1, 2, 3} can be reached. By using R computed above along with (1.15), we obtain the F(i, j) for i, j ∈ {6, 7, 8}. The F(i, j) for i, j ∈ {1, 2, 3, 4, 5} follow from (1.13) and (1.14). Finally, for i ∈ {6, 7, 8} and j ∈ {1, 2, 3} we can apply the preceding corollary. The resulting matrix F is

(1.25) E^XAMPLE. Let X be a Markov chain with state space E = {1, 2, . . . , 7} and transition matrix

There are two recurrent classes: C₁ = {1, 2} and C₂ = (3, 4, 5}. Lumping the states of C₁ and C₂ into two absorbing states, we get a new transition matrix

Thus,

Hence the matrix of the probabilities F(i, j) is

(1.26) E^XAMPLE. Let X_n be the total assets (measured in dollars) of a certain firm at the end of the nth year of its existence. If X_n = 0 at some time n, then the firm is said to be bankrupt, and we put X_{n + 1} = X_{n + 2} = · · · = 0. From any other state it is possible to reach (of course, not necessarily in one year) the state 0. Suppose X = {X_n; n ≥ 0} is a Markov chain. By our hypothesis, state 0 is absorbing and all other states are transient, and the transition matrix is in the form

Though we have only one recurrent class, the probability of ever reaching it is not necessarily equal to one (as it would have been if the state space were finite). Putting S = I + Q + · · ·, and g = Sb, we have that

This is not necessarily one, and the probability that the firm never goes bankrupt is (if it was started with an initial capital of i dollars)