  Four dimensional geometric shell shaping (4D-GSS) is introduced as an
approach for closing the nonlinearity-caused shaping gap. This format is
designed at the spectral efficiency of 8 b/4D-sym and is compared against
polarization-multiplexed 16QAM (PM-16QAM) and probabilistically shaped PM-16QAM
(PS-PM-16QAM) in a 400ZR-compatible transmission setup with high amount of
nonlinearities. Reach increase and nonlinearity tolerance are evaluated in
terms of achievable information rates and post-FEC bit-error rate. Numerical
simulations for a single-span, single-channel show that 4D-GSS achieves
increased nonlinear tolerance and reach increase against PM-16QAM and
PS-PM-16QAM when optimized for bit-metric decoding (RBMD). In terms of RBMD,
gains are small with a reach increase of 1.7% compared to PM-16QAM. When
optimizing for mutual information, a larger reach increase of 3% is achieved
compared to PM-16QAM. Moreover, the introduced GSS scheme provides a scalable
framework for designing well-structured 4D modulation formats with low
complexity.
