  DSP (digital signal processing) has been widely applied in optical
communication systems to mitigate signal distortions and has become one of the
key technologies that have sustained data traffic growth over the past decade.
However, the strict energy budget of application-specific integrated
circuit-based DSP chips has prevented the deployment of some powerful but
computationally costly DSP algorithms. As a result, fiber nonlinearity-induced
signal distortions impede fiber communications systems, especially in
wavelength-division multiplexed (WDM) transmission systems. To solve these
challenges, photonics hardware (i.e., photonic neural networks) promises to
break performance limitations in electronics and gain advantages in bandwidth,
latency, and power consumption in solving intellectual tasks that are
unreachable by conventional digital electronic platforms. This work proposes a
photonic recurrent neural network (RNN) capable of simultaneously resolving
dispersion and both intra and inter-channel fiber nonlinearities in multiple
WDM channels in the photonic domain, for the first time to our best knowledge.
Furthermore, our photonic RNN can directly process optical WDM signals in the
photonic domain, avoiding prohibitive energy consumption and speed overhead in
analog to digital converters (ADC). We demonstrate in simulation that our
photonic RNN can process multiple WDM channels simultaneously and achieve a
reduced bit error rate compared to typical DSP algorithms for all WDM channels
in a pulse-amplitude modulation 4-level (PAM4) transmission system, thanks to
its unique capability to address inter-channel fiber nonlinearities. In
addition to signal quality performance, the proposed system also promises to
significantly reduce the power consumption and the latency compared to the
state-of-the-art DSP chips, according to our power and latency analysis.
