Experimental study of MLSE rcleceivers in the presence of narrowband and vestigial sideband optical filtering

We report on experimental investigations of real-time maximum-likelihood sequence estimation (MLSE) in the presence of narrowband optical filtering, using 10.7-Gb/s nonreturn-to-zero ON-OFF keying and a fiber grating filter with 6.25-GHz bandwidth. Compared to standard threshold detection, the MLSE eliminates a 10(-3) error floor due to narrowband optical filtering and results in a 2-dB optical signal-to-noise ratio penalty at a 10(-3) bit-error ratio. Furthermore, we demonstrate the ability of the MLSE to simultaneously compensate for narrowband optical filtering and chromatic dispersion. Finally, we investigate the influence of narrowband filter frequency detuning and show that the well-known effect of increased filtering tolerance given by the vestigial sideband effect observed in standard threshold detection, disappears in the presence of the MLSE.