  In this paper, a new methodology is proposed that allows for the
low-complexity development of neural network (NN) based equalizers for the
mitigation of impairments in high-speed coherent optical transmission systems.
In this work, we provide a comprehensive description and comparison of various
deep model compression approaches that have been applied to feed-forward and
recurrent NN designs. Additionally, we evaluate the influence these strategies
have on the performance of each NN equalizer. Quantization, weight clustering,
pruning, and other cutting-edge strategies for model compression are taken into
consideration. In this work, we propose and evaluate a Bayesian
optimization-assisted compression, in which the hyperparameters of the
compression are chosen to simultaneously reduce complexity and improve
performance. In conclusion, the trade-off between the complexity of each
compression approach and its performance is evaluated by utilizing both
simulated and experimental data in order to complete the analysis. By utilizing
optimal compression approaches, we show that it is possible to design an
NN-based equalizer that is simpler to implement and has better performance than
the conventional digital back-propagation (DBP) equalizer with only one step
per span. This is accomplished by reducing the number of multipliers used in
the NN equalizer after applying the weighted clustering and pruning algorithms.
Furthermore, we demonstrate that an equalizer based on NN can also achieve
superior performance while still maintaining the same degree of complexity as
the full electronic chromatic dispersion compensation block. We conclude our
analysis by highlighting open questions and existing challenges, as well as
possible future research directions.
