Information theory

The basic concepts of this theory were introduced in the forties of the twentieth century by K. Shannon. Information theory Is the science of the processes of transmission, storage and retrieval of data in natural, technical and social systems. In many applied scientific fields, such as computer science, linguistics, cryptography, management theory, image processing, genetics, psychology, economics, production organization, the methods of this science are used.

In modern conditions, the theory of information is closely connected with coding theory, which deals with the most common problems of correspondence between signal and message, and the theory of signal processing, in which quantization and signal reconstruction are studied, as well as spectral and correlation analysis of such.

Information theory considers the basic concept of "information" for the most part from the quantitative side, not taking into account its value, and sometimes the meaning. With this approach, the page of the text will contain approximately the same amount of data, defined only by the number of characters and symbols, and not depending on what is actually printed there, even if absolutely meaningless and chaotic collection of certain symbols.

This approach is legitimate, but only for modeling communication systems, since they must accurately transmit information through communication channels, which can be represented by any set of signs and symbols. At the same time, when it is necessary to take into account the value and meaning of the data, the quantitative approach is unacceptable. Such circumstances impose significant limitations on the areas of possible application of this theory.

The fundamentals of information theory involve the consideration of various issues, including those directly related to the transmission and reception of data. The basic scheme of communication, which is considered in the study, is as follows. Theory of information and coding Believes that the data is created by a message source, which is a word or a set of words written in the letters of a certain alphabet. The source of the message can be text in any natural or artificial languages, human speech, databases and some mathematical models that create a sequence of letters. The transmitter converts the message into a signal corresponding to the physical nature of the communication channel-the medium for signal transmission. During the passage of such a noise, which introduces distortions into the values of information parameters, may act on it. The receiver restores the original message according to the signal received with distortion. The message in the restored form comes to the addressee - to some person or technical device.

The source of the message is of a statistical nature, that is, the appearance of each message is determined by some probability. Information theory Shannon believes that if the probability of the appearance of the message is equal to one, that is, its appearance is reliable and there is no uncertainty, believe that it does not carry any information.

One of the important problems of information theory is the matching of the communication channel and information properties of the source of messages. Bandwidth is determined by a unit of measurement of 1 bit per second.

One of the problems of communication systems is interference in the path of a useful signal. Shannon's theorem, unfortunately, does not give us a concrete method of dealing with such. The simplest method of elimination, which consists in repeated repetition of messages, is not very effective, since it takes a lot of time to transfer information. Much more effective is the use of codes that can detect and correct errors in the transmission of information.