Convolutional long short-term memory neural network equalizer for nonlinear Fourier transform-based optical transmission systems

<jats:p>We evaluate improvement in the performance of the optical transmission systems operating with the continuous nonlinear Fourier spectrum by the artificial neural network equalisers installed at the receiver end. We propose here a novel equaliser designs based on bidirectional long short-term memory (BLSTM) gated recurrent neural network and compare their performance with the equaliser based on several fully connected layers. The proposed approach accounts for the correlations between different nonlinear spectral components. The application of BLSTM equaliser leads to a 16x improvement in terms of bit-error rate (BER) compared to the non-equalised case. The proposed equaliser makes it possible to reach the data rate of 170 Gbit/s for one polarisation conventional nonlinear Fourier transform (NFT) based system at 1000 km distance. We show that our new BLSTM equalisers significantly outperform the previously proposed scheme based on a feed-forward fully connected neural network. Moreover, we demonstrate that by adding a 1D convolutional layer for the data pre-processing before BLSTM recurrent layers, we can further enhance the performance of the BLSTM equaliser, reaching 23x BER improvement for the 170 Gbit/s system over 1000 km, staying below the 7% forward error correction hard decision threshold (HD-FEC).</jats:p>