  Faster-than-Nyquist (FTN) signaling is a candidate non-orthonormal
transmission technique to improve the spectral efficiency (SE) of future
communication systems. However, such improvements of the SE are at the cost of
additional computational complexity to remove the intentionally introduced
intersymbol interference. In this paper, we investigate the use of deep
learning (DL) to reduce the detection complexity of FTN signaling. To eliminate
the need of having a noise whitening filter at the receiver, we first present
an equivalent FTN signaling model based on using a set of orthonormal basis
functions and identify its operation region. Second, we propose a DL-based list
sphere decoding (DL-LSD) algorithm that selects and updates the initial radius
of the original LSD to guarantee a pre-defined number $N_{\text{L}}$ of lattice
points inside the hypersphere. This is achieved by training a neural network to
output an approximate initial radius that includes $N_{\text{L}}$ lattice
points. At the testing phase, if the hypersphere has more than $N_{\text{L}}$
lattice points, we keep the $N_{\text{L}}$ closest points to the point
corresponding to the received FTN signal; however, if the hypersphere has less
than $N_{\text{L}}$ points, we increase the approximate initial radius by a
value that depends on the standard deviation of the distribution of the output
radii from the training phase. Then, the approximate value of the
log-likelihood ratio (LLR) is calculated based on the obtained $N_{\text{L}}$
points. Simulation results show that the computational complexity of the
proposed DL-LSD is lower than its counterpart of the original LSD by orders of
magnitude.
