Deep Learning for Communication over Dispersive Nonlinear Channels: Performance and Comparison with Classical Digital Signal Processing

In this paper we focus on the application of deep learning for communication over dispersive nonlinear channels, as encountered in low-cost optical intensity modulation/direct detection (IM/DD) links. We apply key enhancements to a state-of-the-art deep learning-based scheme for the optical IM/DD, namely the recently proposed sliding window bidirectional recurrent neural network (SBRNN) autoencoder. We show that its performance can be improved by optimizing the weightings in the sequence estimation algorithm at the receiver. Moreover, additional reduction in the bit-error rate (BER) of the system is achieved by performing bit-to-symbol mapping optimization. Furthermore, we carry out a detailed comparison with classical schemes based on pulse-amplitude modulation and maximum likelihood sequence estimation, a receiver processing provenly optimal for communication over dispersive channels. Our investigation shows that for a reference 42,Gb/s transmission, the SBRNN autoencoder achieves BER performance close to the benchmark system, accounting for the same amount of memory. Crucially, it requires fewer multiplications per decoded bit, thus achieving near-optimal performance at much lower computational complexity.