The Joint Optimization of Transmitted Signal and Receiving Filter for Data Transmission Systems

In most instances a data system designer is faced with the problem of transmitting through a noisy channel over which he has no control. In other words, the designer must concern himself with transmitter and receiver terminals which reduce the disturbing effects of the channel. Nyquist's classic paper1 considered over-all system designs which eliminate intersymbol interference. Others (see, for example, Ref. 2) have also reduced the effects of noise by designing the transmitter and receiver to minimize the noise output for a given Nyquist characteristic and an ideal channel. Tufts' recent work3-6 has recognized that transmission without intersymbol interference may not be the most desirable. He has considered the problem of optimizing transmitted signal waveforms or receiver filters under the criterion of minimizing the mean square error (thus minimizing the joint contribution of noise and inter2363 2364 THE BELL SYSTEM TECHNICAL JOURNAL, DECEMBER 19G5 symbol interference). His attempt at joint optimization of the transmitter and receiver was successful only with the added condition that the transmitted waveform be time limited to one bit interval. This is an important case, but the unsolved joint optimization problem without this constraint is also important. The joint optimization solution would, in effect, provide a performance bound for a given channel. This paper extends Tufts' results by solving the joint optimization problem (mean square error criterion) for a time-invariant transmitter and receiver subject only to an average power constraint on the transmitted waveform.