  The nondispersive per-sample channel model for the optical fiber channel is
considered. Under certain smoothness assumptions, the problem of finding the
minimum amount of noise energy that can render two different input points
indistinguishable is formulated. This minimum noise energy is then taken as a
measure of distance between the points in the input alphabet. Using the
machinery of optimal control theory, necessary conditions that describe the
minimum-energy noise trajectories are stated as a system of nonlinear
differential equations. It is shown how to find the distance between two input
points by solving this system of differential equations. The problem of
designing signal constellations with the largest minimum distance subject to a
peak power constraint is formulated as a clique-finding problem. As an example,
a 16-point constellation is designed and compared with conventional quadrature
amplitude modulation. A computationally efficient approximation for the
proposed distance measure is provided. It is shown how to use this
approximation to design large constellations with large minimum distances.
Based on the control-theoretic viewpoint of this paper, a new decoding scheme
for such nonlinear channels is proposed.
