  Considering a heterogeneous network (HetNet) system consisting of a macro
tier overlaid with a second tier of small cells (SCs), this paper studies the
mean square error (MSE) based precoding design to be employed by the macro base
station and the SC nodes for multiple-input multiple-output (MIMO) downlinks.
First, a new sum-MSE of all users based minimization problem is proposed aiming
to design a set of macro cell (MC) and SC transmit precoding matrices or
vectors. To solve it, two different algorithms are presented. One is via a
relaxed-constraints based alternating optimization (RAO) realized by efficient
alternating optimization and relaxing non-convex constraints to convex ones.
The other is via an unconstrained alternating optimization with normalization
(UAON) implemented by introducing the constraints into the iterations with the
normalization operation. Second, a separate MSE minimization based precoding is
proposed by considering the signal and interference terms corresponding to the
macro tier and the SCs separately. Simulation results show that the sum-MSE
based RAO algorithm provides the best MSE performance among the proposed
schemes under a number of system configurations. When the number of antennas at
the macro-BS is sufficiently large, the MSE of the separate MSE-based precoding
is found to approach that of RAO and surpass that of UAON. Together, this paper
provides a suite of three new precoding techniques that is expected to meet the
need in a broad range of HetNet environments with adequate balance between
performance and complexity.
