  Hybrid transceiver can strike a balance between complexity and performance of
multiple-input multiple-output (MIMO) systems. In this paper, we develop a
unified framework on hybrid MIMO transceiver design using matrix-monotonic
optimization. The proposed framework addresses general hybrid transceiver
design, rather than just limiting to certain high frequency bands, such as
millimeter wave (mmWave) or terahertz bands or relying on the sparsity of some
specific wireless channels. In the proposed framework, analog and digital parts
of a transceiver, either linear or nonlinear, are jointly optimized. Based on
matrix-monotonic optimization, we demonstrate that the combination of the
optimal analog precoders and processors are equivalent to eigenchannel
selection for various optimal hybrid MIMO transceivers. From the optimal
structure, several effective algorithms are derived to compute the analog
transceivers under unit modulus constraints. Furthermore, in order to reduce
computation complexity, a simple random algorithm is introduced for analog
transceiver optimization. Once the analog part of a transceiver is determined,
the closed-form digital part can be obtained. Numerical results verify the
advantages of the proposed design.
