The Transmission Distortion of a Source as a Function of the Encoding Block Length

By now the results originally obtained by Shannon 1 relating reliability and channel capacity are well known. Roughly speaking, they state that perfect transmission can be achieved if, and only if, the capacity of the channel in the transmission link is greater than the information content of the source. For amplitude and time discrete sources the information content is the entropy of the source, but for amplitude continuous sources the entropy and the information content are not the same since the information content is infinite. This, of course, implies that perfect transmission of amplitude continuous sources, or discrete sources with an entropy t h a t is "too large," is impossible with a given finite capacity channel. Yet this is just the situation that is often presented to the communication engineer who must then try to reduce the average distortion to the lowest possible, or practicable, level. For communication systems in which the capacity of the channel is not sufficient to allow perfect transmission, there are two obvious questions to ask: (i) How small can the average distortion be made if any transmission strategy at all is allowed? (ii) How much does the system complexity, or cost, increase when you are required to get "closer" to this minimum? To answer the first question, Shannon generalized his results in a later paper 2 in which the channel requirements are found t h a t are necessary and sufficient to allow transmission at a given level of distortion, or a given error rate.