  Hybrid analog and digital beamforming (HBF) has recently emerged as an
attractive technique for millimeter-wave (mmWave) communication systems. It
well balances the demand for sufficient beamforming gains to overcome the
propagation loss and the desire to reduce the hardware cost and power
consumption. In this paper, the mean square error (MSE) is chosen as the
performance metric to characterize the transmission reliability. Using the
minimum sum-MSE criterion, we investigate the HBF design for broadband mmWave
transmissions. To overcome the difficulty of solving the multi-variable design
problem, the alternating minimization method is adopted to optimize the hybrid
transmit and receive beamformers alternatively. Specifically, a manifold
optimization based HBF algorithm is firstly proposed, which directly handles
the constant modulus constraint of the analog component. Its convergence is
then proved. To reduce the computational complexity, we then propose a
low-complexity general eigenvalue decomposition based HBF algorithm in the
narrowband scenario and three algorithms via the eigenvalue decomposition and
orthogonal matching pursuit methods in the broadband scenario. A particular
innovation in our proposed alternating minimization algorithms is a carefully
designed initialization method, which leads to faster convergence.
{Furthermore, we extend the sum-MSE based design to that with weighted sum-MSE,
which is then connected to the spectral efficiency based design.} Simulation
results show that the proposed HBF algorithms achieve significant performance
improvement over existing ones, and perform close to full-digital beamforming.
