  We developed machine learning approaches for data-driven trellis-based soft
symbol detection in coded transmission over intersymbol interference (ISI)
channels in presence of bursty impulsive noise (IN), for example encountered in
wireless digital broadcasting systems and vehicular communications. This
enabled us to obtain optimized detectors based on the Bahl-Cocke-Jelinek-Raviv
(BCJR) algorithm while circumventing the use of full channel state information
(CSI) for computing likelihoods and trellis state transition probabilities.
First, we extended the application of the neural network (NN)-aided BCJR,
recently proposed for ISI channels with additive white Gaussian noise (AWGN).
Although suitable for estimating likelihoods via labeling of transmission
sequences, the BCJR-NN method does not provide a framework for learning the
trellis state transitions. In addition to detection over the joint ISI and IN
states we also focused on another scenario where trellis transitions are not
trivial: detection for the ISI channel with AWGN with inaccurate knowledge of
the channel memory at the receiver. Without access to the accurate state
transition matrix, the BCJR- NN performance significantly degrades in both
settings. To this end, we devised an alternative approach for data-driven BCJR
detection based on the unsupervised learning of a hidden Markov model (HMM).
The BCJR-HMM allowed us to optimize both the likelihood function and the state
transition matrix without labeling. Moreover, we demonstrated the viability of
a hybrid NN and HMM BCJR detection where NN is used for learning the
likelihoods, while the state transitions are optimized via HMM. While reducing
the required prior channel knowledge, the examined data-driven detectors with
learned trellis state transitions achieve bit error rates close to the optimal
full CSI-based BCJR, significantly outperforming detection with inaccurate CSI.
