  We study the design of amplitude phase-shift keying (APSK) constellations for
a coherent fiber-optical communication system where nonlinear phase noise
(NLPN) is the main system impairment. APSK constellations can be regarded as a
union of phase-shift keying (PSK) signal sets with different amplitude levels.
A practical two-stage (TS) detection scheme is analyzed, which performs close
to optimal detection for high enough input power. We optimize APSK
constellations with 4, 8, and 16 points in terms of symbol error probability
(SEP) under TS detection for several combinations of input power and fiber
length. Our results show that APSK is a promising modulation format in order to
cope with NLPN. As an example, for 16 points, performance gains of 3.2 dB can
be achieved at a SEP of 10^-2 compared to 16-QAM by choosing an optimized APSK
constellation. We also demonstrate that in the presence of severe nonlinear
distortions, it may become beneficial to sacrifice a constellation point or an
entire constellation ring to reduce the average SEP. Finally, we discuss the
problem of selecting a good binary labeling for the found constellations. For
the class of rectangular APSK a labeling design method is proposed, resulting
in near-optimal bit error probability.
