  Coherent optical multi-carrier communications have recently dominated
metro-regional and long-haul optical communications. However, the major
obstacle of networks involving coherent multi-carrier signals such as coherent
optical orthogonal frequency-division multiplexing (CO-OFDM) is the
fiber-induced nonlinearity and the parametric noise amplification from cascaded
optical amplifiers which results in significant nonlinear distortion among
subcarriers. Here, we present the first nonlinear equalizer in optical
communications using the traditional Density-Based Spatial Clustering of
Applications with Noise (DBSCAN) algorithm and a novel modified version of
DBSCAN which combines K-means clustering on the noisy un-clustered symbols. For
a 24.72 Gbit/sec differential quaternary phase-shift keying (DQPSK) CO-OFDM
system, the modified DBSCAN can increase the signal quality-factor by up to
2.158 dB compared to linear equalization at 500 km of transmission. The
modified DBSCAN slightly outperforms the traditional DBSCAN, fuzzy-logic
C-means, hierarchical and conventional K-means clustering at high launched
optical powers.
