  We consider the problem of designing demodulators for linear vector channels
with memory that use reduced-size trellis descriptions for the received signal.
We assume an overall iterative receiver, and for the parts of the signal not
covered by the trellis description, we use interference cancelation based on
the soft information provided by the outer decoder. In order to reach a trellis
description, a linear filter is applied as front-end to compress the signal
structure into a small trellis. This process requires three parameters to be
designed: (i) the front-end filter, (ii) the feedback filter through which the
interference cancelation is done, and (iii) a target response which specifies
the trellis. Demodulators of this form have been studied before under then name
channel shortening (CS), but the interplay between CS and interference
cancelation has not been adequately addressed in the literature. In this paper,
we analyze two types of CS demodulators that are based on the Forney and
Ungerboeck detection models, respectively. The parameters are jointly optimized
based on a generalized mutual information (GMI) function. We also introduce a
third type of CS demodulator that is in general suboptimal but has closed form
solutions for all parameters. Moreover, signal to noise ratio (SNR) asymptotic
properties are analyzed and we show that the third CS demodulator
asymptotically converges to the optimal CS demodulator in the sense of
maximizing the GMI.
