  This paper proposes the design of robust transceivers with
Tomlinson-Harashima precoding (THP) for multiple-input multiple-output (MIMO)
relay systems with amplify-and-forward (AF) protocols based on a multi-branch
(MB) strategy. The MB strategy employs successive interference cancellation
(SIC) on several parallel branches which are equipped with different ordering
patterns so that each branch produces transmit signals by exploiting a certain
ordering pattern. For each parallel branch, the proposed robust nonlinear
transceiver design consists of THP at the source along with a linear precoder
at the relay and a linear minimum-mean-squared-error (MMSE) receiver at the
destination. By taking the channel uncertainties into account, the source and
relay precoders are jointly optimised to minimise the mean-squared-error (MSE).
We then employ a diagonalization method along with some attributes of
matrix-monotone functions to convert the optimization problem with matrix
variables into an optimization problem with scalar variables. We resort to an
iterative method to obtain the solution for the relay and the source precoders
via Karush-Kuhn-Tucker (KKT) conditions. An appropriate selection rule is
developed to choose the nonlinear transceiver corresponding to the best branch
for data transmission. Simulation results demonstrate that the proposed MB-THP
scheme is capable of alleviating the effects of channel state information (CSI)
errors and improving the robustness of the system.
