  In this paper, robust transceiver design with Tomlinson-Harashima precoding
(THP) for multi-hop amplify-and-forward (AF) multiple-input multiple-output
(MIMO) relaying systems is investigated. At source node, THP is adopted to
mitigate the spatial intersymbol interference. However, due to its nonlinear
nature, THP is very sensitive to channel estimation errors. In order to reduce
the effects of channel estimation errors, a joint Bayesian robust design of THP
at source, linear forwarding matrices at relays and linear equalizer at
destination is proposed. With novel applications of elegant characteristics of
multiplicative convexity and matrix-monotone functions, the optimal structure
of the nonlinear transceiver is first derived. Based on the derived structure,
the transceiver design problem reduces to a much simpler one with only scalar
variables which can be efficiently solved. Finally, the performance advantage
of the proposed robust design over non-robust design is demonstrated by
simulation results.
