The maximum error probability E is calculated as follows:
E= max
1≤ j≤mP(incorrect decoding | sj was intended)
= max
1≤ j≤mP(the received w ∈ Bdoes not lie in Nj | wj was transmitted)
= 1 − min
1≤ j≤mP(w ∈ Nj | wjwas transmitted).
Notice that, for each j , P(w ∈ Nj | wj was transmitted) is the quantity mul-tiplied by fj in the expression for R, above. For instance, again referring to the circumstances of Examples 4.5.2 and 4.5.3, we calculate that c has the least likelihood, 0.828, of being correctly transmitted, and thus, for that code and channel, E= 1 − .828 = .172.
Observe that E does not take the source frequencies into account (although they do enter anyway, in the construction of the MLD table). It is a “worst-case”
sort of measure of error likelihood.
4.5.8 For many code-and-channel systems we have “do not decode” occurring in the right hand column (the decode column) of the MLD table, corresponding to wordsw ∈ B for which there are two or more j for which P(wj,w) is maximal. With such a table, we have a number of choices to make in assessing the likelihood of error. Should a “do not decode” message, which surely signals
some sort of failure of the system, weigh as much in our estimation as an out-and-out error, in which we decode the wrong source message from the received wordw? In the definitions of E and E, above, the two different sorts of error are treated as the same, but most people would agree that in most situations the “do not decode” message is a less serious sort of error than an incorrect decoding of which we are unaware.
There are an endless number of ways of weighting the significance of the various errors that could occur, using any particular code-and-channel system.
You should be aware that the definitions of E and E given here are not graven in stone, and that in the real world you might do well to fashion a measure of error likelihood appropriate to the real situation, a measure which takes your weighting of error significance into account. See the end of this section for exercises on error weighting, and on the computation of reliability and error when NCWD is used.
Reliability of a channel Suppose we have a channel with input alphabet A, output alphabet B, and transition probabilities qi j, i= 1,...,n, j = 1,...,k. Let us adjoin a code by taking S= A, fj= pj, j= 1,...,n, where (p1,..., pn) is some n-tuple of optimal input frequencies, and the encoding scheme aj→ aj, j= 1,...,n. The reliability R (with respect to MLD) of the resulting code-and-channel system will be called the reliability of the code-and-channel. (Perhaps the definite article is not justified here when(p1,..., pn) is not unique; we pass over this difficulty for now.)
In the case of an n-ary symmetric channel, one satisfying the hypothesis of Theorem 4.5.5, we can take(p1,..., pn) = (1/n,...,1/n), and then the equal-ity of the imposed source frequencies f1,..., fnimplies that MLD and NCWD coincide, by Theorem 4.5.5. Clearly, for each aj∈ A, the unique word over A of length 1 closest to aj is aj itself. That is, Nj= {aj}. Thus
R=1 n
n j=1
P(aj is received | aj is sent)
=1 n
n j=1
qj j=1
nnq= q,
the constant main diagonal entry in the matrix of transition probabilities.
Consequently, the definition of channel reliability given here agrees with the prior definition of the reliability of an n-ary symmetric channel, at least when that reliability is greater than 1/n.
When the optimal input frequencies are difficult to obtain, and the channel is “close” to being n-ary symmetric (see the discussion insection 3.4), a rough estimate of the reliability of the channel may be obtained by taking the source frequencies to be equal (to 1/n). For instance, consider the channel described in Example 4.5.2. With respect to the the encoding scheme 0→ 0, 1 → 1, we
4.5 Error correction and reliability 103
have N0= {0,∗} and N1= {1} (since P(0,∗) = 12(.04) > P(1,∗) = 12(.03)), whence R≈12[.9 + .04] +12[.92] = .93.
The reliability of a discrete memoryless channel, as defined here, appears to be a new index of channel quality. Its relation to channel capacity has not been worked out, and it is not yet clear what role, if any, reliability will play in the theory of communication.
Exercises 4.5
1. In each of the following, you are given a source alphabet S, a code (and input) alphabet A, an output alphabet B, source frequencies f1,..., fm, an encoding scheme, and the matrix Q of transition probabilities. In each case, produce (i) an MLD table, (ii) the reliability R, and (iii) the maximum error probability of the code-and-channel system. e→ 100, and the channel is binary symmetric with reliability .9.
2. Suppose the available channel is binary symmetric with reliability .8.
Suppose S = {a,b}, fa = .999, fb= .001, and the encoding scheme is a→ 000,b → 111.
(a) Verify that MLD will decode every wordw ∈ {0,1}3as ‘a’.
(b) Calculate the reliability of this code-and-channel system and the max-imum error probability.
(c) Same question as (b), but use NCWD.
(d) How large must t be so that, if we consider the encoding scheme a→ 0t= 0···0 (t zeroes) and b → 1t, then MLD will decode 1t as b?
(e) Find the reliability and the maximum error probability of the code-and-channel system obtained by taking the scheme you found in part (d), when the decoding method is MLD and again when it is NCWD.
3. For each instance ofδ and ρ as in 4.5.4, and each code-and-channel system with A= B and a fixed-length encoding scheme sj → wj, j= 1,...,m
of length, we can define a nearest-code-word sort of decoding associated withδ and ρ, to be denoted NCWD(δ,ρ), as NCWD was defined, but with the metric dG H arising fromδ and ρ playing the role that dH plays in the definition of NCWD. That is, having receivedw ∈ A, we decodew as that sj for which dG H(w,wj) is the least, provided there is a unique such j. If there is no unique such j , we report “do not decode.”
One reason for considering the metrics dG H is the possibility that, given a code-and-channel system, there may be choice of δ and ρ such that NCWD(δ,ρ) and MLD coincide for that system. The requirements on the system for the existence of such a pair(δ,ρ) await disclosure, but we can see readily that there are cases when there is no such pair.
(a) Show that, with(δ,ρ) and NCWD(δ,ρ) as above, if the code words wj, j= 1,...,m, are distinct, then NCWD(δ,ρ) will decode wj as sj. (b) Conclude that there is no pair(δ,ρ) for which NCWD(δ,ρ) and MLD
coincide on the code-and-channel system of Exercise 2(a), above.
4. Estimate the reliability of each of the channels mentioned in exercise 1, above, by imposing equal source frequencies on the input characters. (Of course, in (c) the result will be exact.)
5. The average error probability can be thought of as the average cost of an attempted transmission of a source letter (from the choosing of the source letter to the result after decoding), where the cost of an error is one unit, and the cost of no error is zero.
It follows that we can refine the average error probability as a measure of system failure by distinguishing more finely among the outcomes of the “choose sj – transmit wj – decode” experiment, assigning different appropriate costs to these outcomes, and then calculating average cost. This type of refinement was alluded to in 4.5.8.
For example, in the circumstances of Exercise 2 above, it may be that source message b is extremely grave and that it would be very costly to mistake a for b. Let us suppose that, whatever the decoding method, de-coding b when a was intended costs 1000 units, dede-coding a when b was intended costs 100 units, getting “do not decode” when a was intended costs one unit, and “do not decode” when b was intended costs 50 units. A correct transmission costs nothing. Find the average cost of an attempted transmission when
(a) the decoding method is MLD and the scheme is as in Exercise 2(a), above;
(b) the decoding method is NCWD and the scheme is as in 2(a);
(c) the decoding method is MLD and the scheme is the one you found in 2(d);
(d) the decoding method is NCWD and the scheme is that of 2(d).
4.5 Error correction and reliability 105
6. In each part of Exercise 1, compute the reliability, the average error proba-bility, and the maximum error probability of the system in which decoding is by NCWD and in which
(i) “do not decode” counts as an error;
(ii) “do not decode” does not count as an error;
(iii) “do not decode” counts as one-half of an error.
In each case, assume that the relative source frequencies are known, and are as given.
7. The channel is binary and symmetric, with reliability p, 1/2 < p < 1.
There are two source messages, a and b, with equal frequency. For a pos-itive integer t, consider the encoding scheme a→ 0t, b→ 1t. Let R(t) and E(t) = 1 − R(t) denote the reliability and average error probability, respectively, of this code-and-channel system, using MLD (= NCWD, in this case); note that E(t) = E(t) by the symmetry of the situation. Let R0(t) and E0(t) denote the corresponding probabilities if “do not decode”
is considered a success, not a failure.
(a) Express R(t) and R0(t) explicitly as functions of p and t.
(b) Show that R(t +1) = R0(t)− t
t/2
pt/2 (1− p)t/2+1for each pos-itive integer t.
(c) Show that R(t) ≤ R(t +2) and that R0(t) ≤ R0(t +2) for each positive integer t.
(d) Show that R(t) → 1 as t → ∞.
[Hints: consider the cases where t is odd or even separately, for (a), (b), and (c). For (d), use the fact that p> 1/2, and the Law of Large Numbers;
seesection 1.9.]
8. Suppose that|S| = m = 35, and we have a shortest possible fixed-length encoding scheme for a uniquely decodable code. How long will the MLD table be when
(a) |A| = |B| = 2;
(b) |A| = 2, |B| = 3;
(c) |A| = |B| = 3?
9. In Exercises 1 (b) and (c), above, note that the encoding schemes are as short as possible, but not thoughtfully conceived. For instance, in 1(b), the code word for e, the most commonly encountered source letter, is a Hamming distance 2 from the word for c, the least common code word, but a distance 1 from each of the words for b and d. Surely it would increase the reliability R if we interchange the code words representing c and d, or c and b.
Verify that this is so. Also, find a fixed-length scheme, of length 3, to replace the scheme in 1(c), which increases the reliability.