  Advanced channel decoders rely on soft-decision decoder inputs for which
mutual information (MI) is the natural figure of merit. In this paper, we
analyze an optical fiber system by evaluating MI as the maximum achievable rate
of transmission of such a system. MI is estimated by means of histograms for
which the correct bin number is determined in a blind way. The MI estimate
obtained this way shows excellent accuracy in comparison with the true MI of
16-state quadrature amplitude modulation (QAM) over an additive white Gaussian
noise channel with additional phase noise, which is a simplified model of a
nonlinear optical fiber channel. We thereby justify to use the MI estimation
method to accurately estimate the MI of an optical fiber system. In the second
part of this work, a transoceanic fiber system with 6000 km of standard
single-mode fiber is simulated and its MI determined. Among rectangular QAMs,
16-QAM is found to be the optimal modulation scheme for this link as to
performance in terms of MI and requirements on components and digital signal
processing. For the reported MI of 3.1 bits/symbol, a minimum coding overhead
of 29% is required when the channel memory is not taken into account. By
employing ideal single-channel digital back-propagation, an increase in MI by
0.25 bits/symbol and 0.28 bits/symbol is reported for 16-QAM and 64-QAM,
respectively, lowering the required overhead to 19% and 16%. When the channel
spacing is decreased to be close to the Nyquist rate, the dual-polarization
spectral efficiency is 5.7 bits/s/Hz, an increase of more than 2 bits/symbol
compared to a 50 GHz spacing.
