  In massive multiple-input multiple-output (MIMO) systems, hybrid
analog-digital (AD) beamforming can be used to attain a high directional gain
without requiring a dedicated radio frequency (RF) chain for each antenna
element, which substantially reduces both the hardware costs and power
consumption. While massive MIMO transceiver design typically relies on the
conventional mean-square error (MSE) criterion, directly minimizing the symbol
error rate (SER) can lead to a superior performance. In this paper, we first
mathematically formulate the problem of hybrid transceiver design under the
minimum SER (MSER) optimization criterion and then develop a MSER-based
gradient descent (GD) iterative algorithm to find the related stationary
points. We then propose a deep-unfolding neural network (NN), in which the
iterative GD algorithm is unfolded into a multi-layer structure wherein a set
of trainable parameters are introduced for accelerating the convergence and
enhancing the overall system performance. To implement the training stage, the
relationship between the gradients of adjacent layers is derived based on the
generalized chain rule (GCR). The deep-unfolding NN is developed for both
quadrature phase shift keying (QPSK) and for $M$-ary quadrature amplitude
modulated (QAM) signals and its convergence is investigated theoretically.
Furthermore, we analyze the transfer capability, computational complexity, and
generalization capability of the proposed deep-unfolding NN. Our simulation
results show that the latter significantly outperforms its conventional
counterpart at a reduced complexity.
