  Bidirectional recurrent neural networks (bi-RNNs), in particular,
bidirectional long short term memory (bi-LSTM), bidirectional gated recurrent
unit, and convolutional bi-LSTM models have recently attracted attention for
nonlinearity mitigation in fiber-optic communication. The recently adopted
approaches based on these models, however, incur a high computational
complexity which may impede their real-time functioning. In this paper, by
addressing the sources of complexity in these methods, we propose a more
efficient network architecture, where a convolutional neural network encoder
and a unidirectional many-to-one vanilla RNN operate in tandem, each best
capturing one set of channel impairments while compensating for the
shortcomings of the other. We deploy this model in two different receiver
configurations. In one, the neural network is placed after a linear
equalization chain and is merely responsible for nonlinearity mitigation; in
the other, the neural network is directly placed after the chromatic dispersion
compensation and is responsible for joint nonlinearity and polarization mode
dispersion compensation. For a 16-QAM 64 GBd dual-polarization optical
transmission over 14x80 km standard single-mode fiber, we demonstrate that the
proposed hybrid model achieves the bit error probability of the
state-of-the-art bi-RNN-based methods with greater than 50% lower complexity,
in both receiver configurations.
