  We consider joint transceiver design for general Multiple-Input
Multiple-Output communication systems that implement interference
(pre-)subtraction, such as those based on Decision Feedback Equalization (DFE)
or Tomlinson-Harashima precoding (THP). We develop a unified framework for
joint transceiver design by considering design criteria that are expressed as
functions of the Mean Square Error (MSE) of the individual data streams. By
deriving two inequalities that involve the logarithms of the individual MSEs,
we obtain optimal designs for two classes of communication objectives, namely
those that are Schur-convex and Schur-concave functions of these logarithms.
For Schur-convex objectives, the optimal design results in data streams with
equal MSEs. This design simultaneously minimizes the total MSE and maximizes
the mutual information for the DFE-based model. For Schur-concave objectives,
the optimal DFE design results in linear equalization and the optimal THP
design results in linear precoding. The proposed framework embraces a wide
range of design objectives and can be regarded as a counterpart of the existing
framework of linear transceiver design.
