  End-to-end learning has become a popular method to optimize a constellation
shape of a communication system. When the channel model is differentiable,
end-to-end learning can be applied with conventional backpropagation algorithm
for optimization of the shape. A variety of optimization algorithms have also
been developed for end-to-end learning over a non-differentiable channel model.
In this paper, we compare gradient-free optimization method based on the
cubature Kalman filter, model-free optimization and backpropagation for
end-to-end learning on a fiber-optic channel modeled by the split-step Fourier
method. The results indicate that the gradient-free optimization algorithms
provide a decent replacement to backpropagation in terms of performance at the
expense of computational complexity. Furthermore, the quantization problem of
finite bit resolution of the digital-to-analog and analog-to-digital converters
is addressed and its impact on geometrically shaped constellations is analysed.
Here, the results show that when optimizing a constellation with respect to
mutual information, a minimum number of quantization levels is required to
achieve shaping gain. For generalized mutual information, the gain is
maintained throughout all of the considered quantization levels. Also, the
results implied that the autoencoder can adapt the constellation size to the
given channel conditions.
