Information theory
From Encyclopædia
Information theory, also called the theory of
communication, is a branch of
probability theory that has been developed to provide a measure of the flow of information from an information source to a destination. It also supplies a measure of the channel capacity of a
communications medium such as a telephone
wire and shows the optimal coding procedures for
communication. Although originally concerned with telephone networks, the theory has a wider application to any
communication process, even as simple as one human being talking to another. It may also be viewed as a branch of CYBERNETICS, the
science of
Control and
communication, and it has strong associations with
Control engineering, theories of learning, and the physiology of the
nervous system.Information theory was developed to a great extent at the
bell Telephone Company laboratories in
New Jersey under the auspices of Claude SHANNON in the 1940s and '50s. Many other versions of the theory have been suggested, notably by D. M. MacKay and Dennis GABOR.Principles.The principal features involved in information theory are a source of information that is encoded and transmitted on a channel to a receiver, where it is decoded.There are two versions of information theory, one for continuous and the other for discrete information systems. The first theory is concerned with the wavelength, amplitude, and
frequency of
communications signals, and the second with the stochastic (random) processes associated with the theory of AUTOMATA. The discrete theory applies to a larger range of applications and was developed for both noiseless and noisy channels. A noisy channel contains unwanted signals and requires a filter to take a copy of the transmitted message and compare it to the message received.
entropy--the Measure of Information.(For a discussion of the Shannon-
weaver measure of information, see this article in the Academic American
encyclopedia.)Channel Capacity.The measure of the channel capacity of an information system is best illustrated where the probabilities again are equal. Given a set of 16 carriers, A, B . . . , P, each carrying 4 bits of information, then the channel capacity is 4n bits per second, where the channel is capable of transmitting n symbols per second; this becomes slightly more complicated when the probabilities are not all the same. The encoding of messages now requires a suitable procedure. It requires
punctuation, as in the
case of a "pause" in Morse code, or alternatively, all the words must be of fixed length. Furthermore, to achieve an optimal code, there are certain procedures that are all based on the principle that the most frequently occurring words (or letters) should be coded with the symbol of shortest duration. Thus e (the most frequently occurring letter in English) would be 1 in binary code, whereas the letter x might be 26 (11010 in binary).Applications.More complicated theorems for continuous and discrete systems, with or without noise, make up the mathematical theory of information. The discrete theory can generate letter sequences and word sequences that can approximate ordinary English. A Markov net is a stochastic process that deals with conditional probabilities. For example, the
probability of q being followed by u in an English word is very nearly 1 (certainty); one can also work out the probabilities for all letters and all words: for instance, the
probability of the being followed by the is very nearly 0 (impossible). Information theory is thus an important tool in the analysis of
language or of any sequence of events--and its encoding, transmission, reception, and decoding. Such methods have been used to describe learning from the point of view of the learner, where the source is one where some pattern of events occurs (in the
case of human learning, this is often nature or life).The theory of information has also been used in some models of the brain, where thoughts and beliefs (some configuration of neurons) are the source; they are encoded in neural
language, translated into a natural
language such as English, and decoded by the hearer into his or her own thoughts. There is also a semantic of information, so far little developed, which deals with meaning, as opposed to uncertainty of information.F. H. GeorgeBibliography: Ash, R. B., Information Theory (1965); Bendat, Julius S., Principles and Applications of Random Noise Theory (1958; repr. 1978); Clark, F., Information Processing (1970); Guiascu, Silviu, Information Theory with New Applications (1977); Haber, Fred, An Introduction to Information and
communication Theory (1974); Kullback, Solomon, Information Theory and Statistics (1974); Littlejohn, Stephen, Theories of Human
communication (1978); MacKay, Donald, Information, Mechanism and Meaning (1970); Meetham, A. R.,
encyclopedia of
linguistics, Information and
Control (1969); Rosie, A. M., Information and
communication Theory, 2d ed. (1973).
