The works of Wiener and Shannon, previously cited, were the beginning of modern statistical communication theory. Both these investigators applied probabilistic methods to the problem of extracting information-bearing signals from noisy backgrounds, but they worked from different standpoints. In this section we briefly examine these two approaches to optimum system design.
1.4.1 Statistical Signal Detection and Estimation Theory
Wiener considered the problem of optimally filtering signals from noise, where optimum is used in the sense of minimizing the average squared error between the desired output and the actual output. The resulting filter structure is referred to as the Wiener filter. This type of approach is most appropriate for analog communication systems in which the demodulated output of the receiver is to be a faithful replica of the message input to the transmitter.
Wieners approach is reasonable for analog communications. However, in the early 1940s, (North, 1943) provided a more fruitful approach to the digital communications problem, in which the receiver must distinguish between a number of discrete signals in background noise. Actually, North was concerned with radar, which requires only the detection of the presence or absence of a pulse. Since fidelity of the detected signal at the receiver is of no consequence in such signal-detection problems, North sought the filter that would maximize the peak-signal-to-root-mean-square (rms) noise ratio at its output. The resulting optimum filter is called the matched filter, for reasons that will become apparent in Chapter 8, where we consider digital data transmission. Later adaptations of the Wiener and matched-filter ideas to time-varying backgrounds resulted in adaptive filters. We will consider a subclass of such filters in Chapter 8 when equalization of digital data signals is discussed.
The signal-extraction approaches of Wiener and North, formalized in the language of statistics in the early 1950s by several researchers [see Middleton (1960), p. 832, for several references], were the beginnings of what is today called statistical signal detection and estimation theory. In considering the design of receivers utilizing all the information available at the channel output, Woodward and Davies (1952) determined that this so-called ideal receiver computes the probabilities of the received waveform given the possible transmitted messages. These computed probabilities are known as a posteriori probabilities. The ideal receiver then makes the decision that the transmitted message was the one corresponding to the largest a posteriori probability. Although perhaps somewhat vague at this point, this maximum a posteriori (MAP) principle, as it is called, is one of the cornerstones of detection and estimation theory. Another development that had far-reaching consequences in the development of detection theory was the application of generalized vector space ideas (Kotelnikov, 1959; Wozencraft and Jacobs, 1965). We will examine these ideas in more detail in Chapters 8 through 10.
1.4.2 Information Theory and Coding
The basic problem that Shannon considered is, Given a message source, how shall the messages produced be represented so as to maximize the information conveyed through a given channel? Although Shannon formulated his theory for both discrete and analog sources, we will think here in terms of discrete systems. Clearly, a basic consideration in this theory is a measure of information. Once a suitable measure has been defined (and we will do so in Chapter 11), the next step is to define the information carrying capacity, or simply capacity, of a channel as the maximum rate at which information can be conveyed through it. The obvious question that now arises is,Given a channel, how closely can we approach the capacity of the channel, and what is the quality of the received message? A most surprising, and the singularly most important, result of Shannons theory is that by suitably restructuring the transmitted signal, we can transmit information through a channel at any rate less than the channel capacity with arbitrarily small error, despite the presence of noise, provided we have an arbitrarily long time available for transmission. This is the gist of Shannons second theorem. Limiting our discussion at this point to binary discrete sources, a proof of Shannons second theorem proceeds by selecting code words at random from the set of 2n possible binary sequences n digits long at the channel input. The probability of error in receiving a given n-digit sequence, when averaged over all possible code selections, becomes arbitrarily small as n becomes arbitrarily large. Thus many suitable codes exist, but we are not told how to find these codes. Indeed, this has been the dilemma of information theory since its inception and is an area of active research. In recent years, great strides have been made in finding good coding and decoding techniques that are implementable with a reasonable amount of hardware and require only a reasonable amount of time to decode. Several basic coding techniques will be discussed in Chapter 11.15Perhaps the most astounding development in the recent history of coding was the invention of turbo coding and subsequent publication by French researchers in 1993.16 Their results, which were subsequently verified by several researchers, showed performance to within a fraction of a decibel of the Shannon limit.17
1.4.3 Recent Advances
There have been great strides made in communications theory and its practical implementation in the past few decades. Some of these will be pointed out later in the book. To capture the gist of these advances at this point would delay the coverage of basic concepts of communications theory, which is the underlying intent of this book. For those wanting additional reading at this point, two recent issues of the IEEE Proceedings will provide information in two areas:
15For a good survey on Shannon theory, as it is known, see S. Verdu, Fifty Years of Shannon Theory, IEEE Trans. Infor. Theory, 44: pp. 2057–2078, Oct., 1998.
16C. Berrou, A. Glavieux, and P. Thitimajshima, Near Shannon Limit Error-Correcting Coding and Decoding: Turbo Codes, Proc. 1993 Int. Conf. Commun., Geneva, Switzerland, 1064–1070, May 1993. See also D. J. Costello and G. D. Forney, Channel Coding: The Road to Channel Capacity, Proc. IEEE, 95: 1150–1177, June 2007 for an excellent tutorial article on the history of coding theory.
17
Actually low-density parity-check codes, invented and published by Robert Gallager in 1963, were the first codes to allow data transmission rates close to the theoretical limit (Gallager, 1963). However, they were impractical to implement in 1963, so were forgotten about until the past 10 to 20 years whence practical advances in their theory and substantially advanced processors have spurred a resurgence of interest in them.
turbo-information processing (used in decoding turbo codes among other applications)18, and multiple-input multiple-output (MIMO) communications theory, which is expected to have far-reaching impact on wireless local- and wide-area network development.19An appreciation for the broad sweep of developments from the beginnings of modern communications theory to recent times can be gained from a collection of papers put together in a single volume, spanning roughly 50 years, that were judged to be worthy of note by experts in the field.20