  The application of hybrid precoding in millimeter wave (mmWave)
multiple-input multiple-output (MIMO) systems has been proved effective for
reducing the number of radio frequency (RF) chains. However, the maximum number
of independent data streams is conventionally restricted by the number of RF
chains, which leads to limiting the spatial multiplexing gain. To further
improve the achievable spectral efficiency (SE), in this paper we propose a
novel generalized spatial modulation (GenSM) aided mmWave MIMO system to convey
an extra data stream via the index of the active antennas group, while no extra
RF chain is required. Moreover, we also propose a hybrid analog and digital
precoding scheme for SE maximization. More specifically, a closed-form lower
bound is firstly derived to quantify the achievable SE of the proposed system.
By utilizing this lower bound as the cost function, a two-step algorithm is
proposed to optimize the hybrid precoder. The proposed algorithm not only
utilizes the concavity of the cost function over the digital power allocation
vector, but also invokes the convex $\ell_\infty$ relaxation to handle the
non-convex constraint imposed by analog precoding. Finally, the proposed scheme
is shown via simulations to outperform state-of-the-art mmWave MIMO schemes in
terms of achievable SE.
