  We investigate methods for experimental performance enhancement of
auto-encoders based on a recurrent neural network (RNN) for communication over
dispersive nonlinear channels. In particular, our focus is on the recently
proposed sliding window bidirectional RNN (SBRNN) optical fiber autoencoder. We
show that adjusting the processing window in the sequence estimation algorithm
at the receiver improves the reach of simple systems trained on a channel model
and applied "as is" to the transmission link. Moreover, the collected
experimental data was used to optimize the receiver neural network parameters,
allowing to transmit 42 Gb/s with bit-error rate (BER) below the 6.7%
hard-decision forward error correction threshold at distances up to 70km as
well as 84 Gb/s at 20 km. The investigation of digital signal processing (DSP)
optimized on experimental data is extended to pulse amplitude modulation with
receivers performing sliding window sequence estimation using a feed-forward or
a recurrent neural network as well as classical nonlinear Volterra
equalization. Our results show that, for fixed algorithm memory, the DSP based
on deep learning achieves an improved BER performance, allowing to increase the
reach of the system.
