  We consider optimal channel shortener design for reduced-state soft-output
Viterbi equalizer (RS-SOVE) in single-carrier (SC) systems. To use RS-SOVE,
three receiver filters need to be designed: a prefilter, a target response and
a feedback filter. The collection of these three filters are commonly referred
to as the \lq\lq{}channel shortener\rq\rq{}. Conventionally, the channel
shortener is designed to transform an intersymbol interference (ISI) channel
into an equivalent minimum-phase equivalent form. In this paper, we design the
channel shortener to maximize a mutual information lower bound (MILB) based on
a mismatched detection model. By taking the decision-feedback quality in the
RS-SOVE into consideration, the prefilter and feedback filter are found in
closed forms, while the target response is optimized via a gradient-ascending
approach with the gradient explicitly derived. The information theoretical
properties of the proposed channel shortener are analyzed. Moreover, we show
through numerical results that, the proposed channel shortener design achieves
superior detection performance compared to previous channel shortener designs
at medium and high code-rates.
