  Training of autoencoders using the back-propagation algorithm is challenging
for non-differential channel models or in an experimental environment where
gradients cannot be computed. In this paper, we study a gradient-free training
method based on the cubature Kalman filter. To numerically validate the method,
the autoencoder is employed to perform geometric constellation shaping on
differentiable communication channels, showing the same performance as the
back-propagation algorithm. Further investigation is done on a
non-differentiable communication channel that includes: laser phase noise,
additive white Gaussian noise and blind phase search-based phase noise
compensation. Our results indicate that the autoencoder can be successfully
optimized using the proposed training method to achieve better robustness to
residual phase noise with respect to standard constellation schemes such as
Quadrature Amplitude Modulation and Iterative Polar Modulation for the
considered conditions.
