  Two-way relaying can improve spectral efficiency in two-user cooperative
communications. It also has great potential in multiuser systems. A major
problem of designing a multiuser two-way relay system (MU-TWRS) is transceiver
or precoding design to suppress co-channel interference. This paper aims to
study linear precoding designs for a cellular MU-TWRS where a multi-antenna
base station (BS) conducts bi-directional communications with multiple mobile
stations (MSs) via a multi-antenna relay station (RS) with amplify-and-forward
relay strategy. The design goal is to optimize uplink performance, including
total mean-square error (Total-MSE) and sum rate, while maintaining individual
signal-to-interference-plus-noise ratio (SINR) requirement for downlink
signals. We show that the BS precoding design with the RS precoder fixed can be
converted to a standard second order cone programming (SOCP) and the optimal
solution is obtained efficiently. The RS precoding design with the BS precoder
fixed, on the other hand, is non-convex and we present an iterative algorithm
to find a local optimal solution. Then, the joint BS-RS precoding is obtained
by solving the BS precoding and the RS precoding alternately. Comprehensive
simulation is conducted to demonstrate the effectiveness of the proposed
precoding designs.
