  Integrated sensing and communications (ISAC) has been considered one of the
new paradigms for sixth-generation (6G) wireless networks. In the
millimeter-wave (mmWave) ISAC system, hybrid beamforming (HBF) is considered an
emerging technology to exploit the limited number of radio frequency (RF)
chains in order to reduce the system hardware cost and power consumption.
However, the HBF structure reduces the spatial degrees of freedom for the ISAC
system, which further leads to increased interference between multiple users
and between users and radar sensing. To solve the above problem, rate split
multiple access (RSMA), which is a flexible and robust interference management
strategy, is considered. We investigate the joint common rate allocation and
HBF design problem for the HBF-based RSMA-assisted mmWave ISAC scheme. We
propose the penalty dual decomposition (PDD) method coupled with the weighted
mean squared error (WMMSE) minimization method to solve this high-dimensional
non-convex problem, which converges to the Karush-Kuhn-Tucker (KKT) point of
the original problem. Then, we extend the proposed algorithm to the HBF design
based on finite-resolution phase shifters (PSs) to further improve the energy
efficiency of the system. Simulation results demonstrate the effectiveness of
the proposed algorithm and show that the RSMA-ISAC scheme outperforms other
benchmark schemes.
