  Hybrid precoding design is challenging for millimeter-wave (mmWave) massive
MIMO. Most prior hybrid precoding schemes are designed to maximize the sum
spectral efficiency (SSE), while seldom investigate the bit-error-rate (BER).
Therefore, we propose an over-sampling codebook (OSC)-based hybrid minimum
sum-mean-square-error (min-SMSE) precoding scheme for mmWave multi-user
three-dimensional (3D)-MIMO systems to optimize the BER, where multi-user
transmission with multiple data streams for each user is considered.
Specifically, given the effective baseband channel consisting of the real
channel and analog precoding, we first design the digital precoder/combiner
based on min-SMSE criterion to optimize the BER. To further reduce the SMSE
between the transmit and receive signals, we propose an OSC-based joint analog
precoder/combiner (JAPC) design. Simulation results show that the proposed
scheme can achieve the better performance than its conventional counterparts.
