  Recently, data-driven approaches motivated by modern deep learning have been
applied to optical communications in place of traditional model-based
counterparts. The application of deep neural networks (DNN) allows flexible
statistical analysis of complicated fiber-optic systems without relying on any
specific physical models. Due to the inherent nonlinearity in DNN, various
equalizers based on DNN have shown significant potentials to mitigate fiber
nonlinearity. In this paper, we propose a turbo equalization (TEQ) based on DNN
as a new alternative framework to deal with nonlinear fiber impairments for
future coherent optical communications. The proposed DNN-TEQ is constructed
with nested deep residual networks (ResNet) to train extrinsic likelihood given
soft-information feedback from channel decoding. Through extrinsic information
transfer (EXIT) analysis, we verify that our DNN-TEQ can accelerate decoding
convergence to achieve a significant gain in achievable throughput by
0.61b/s/Hz. We also demonstrate that optimizing irregular low-density
parity-check (LDPC) codes to match EXIT chart of the DNN-TEQ can improve
achievable rates by up to 0.12 b/s/Hz.
