  This paper studies hybrid beamforming (HB)-based non-orthogonal multiple
access (NOMA) in multiuser millimeter wave (mmWave) communications. HB offers
power-efficient and low-complexity precoding for downlink multiuser mmWave
systems which increases multiplexing gain and spectral efficiency of the
system. Applying NOMA to HB-based systems, called HB-NOMA, can scale the number
of users while offering a high spectral efficiency. However, an imperfect
correlation between the effective channels of users in each NOMA cluster
seriously degrades the achievable rate of HB-NOMA. In this paper, first a
sum-rate maximization problem is formulated for HB-NOMA, and an algorithm is
proposed to solve it effectively. It is then shown that the relationship
between the effective channels of the users in each NOMA cluster can be
approximated by a correlation factor. Next, the effect of imperfect correlation
is analyzed, and a lower bound on the achievable rate of the users is derived
for both perfect and imperfect correlation. Finally, the rate gap resulting
from an imperfect correlation is evaluated and a tight upper bound is derived
for that. Simulation results show that low correlation degrades the achievable
rate of users. The lower bounds are tight in the large dimensional regime and
in single-path channels.
