  Non-orthogonal multiple access (NOMA) and millimeter wave (mmWave) are two
key enabling technologies for the fifth-generation (5G) mobile networks and
beyond. In this paper, we consider uplink communications with a hybrid
beamforming structure and focus on improving the spectral efficiency (SE) and
energy efficiency (EE) of mmWave multiple-input multiple-output (MIMO)-NOMA
systems with enhanced user grouping and power allocation. Exploiting the
directionality feature of mmWave channels, we first propose a novel initial
agglomerative nesting (AGNES) based user grouping algorithm by taking advantage
of the channel correlations. It is noted that the optimization of the SE/EE is
a challenging task due to the non-linear programming nature of the
corresponding problem involving user grouping, beam selection, and power
allocation. Our idea is to decompose the overall optimization problem into a
mixed integer problem comprising of user grouping and beam selection only,
followed by a continuous problem involving power allocation and digital
beamforming design. To avoid the prohibitively high complexity of the
brute-force search approach, we propose two suboptimal low-complexity user
grouping and beam selection schemes, the direct AGNES (DIR-AGNES) scheme and
the successive AGNES (SUC-AGNES) scheme. We also introduce the quadratic
transform (QT) to recast the non-convex power allocation optimization problem
into a convex one subject to a minimum required data rate of each user. The
continuous problem is solved by iteratively optimizing the power and the
digital beamforming. Extensive simulation results have shown that our proposed
mmWave-NOMA design outperforms the conventional orthogonal multiple access
(OMA) scenario and the state-of-art NOMA schemes.
