  Integrated Sensing and Communication (ISAC) emerges as a promising technology
for B5G/6G, particularly in the millimeter-wave (mmWave) band. However, the
widespread adoption of hybrid architecture in mmWave systems compromises
multiplexing gain due to limited radio-frequency chains, resulting in mediocre
performance when embedding sensing functionality. To avoid sacrificing the
spectrum efficiency in hybrid structures while addressing performance
bottlenecks in its extension to ISAC, we present an optimized beam pattern
modulation-embedded ISAC (BPM-ISAC). BPM-ISAC applies index modulation over
beamspace by selectively activating communication beams, aiming to minimize
sensing beampattern mean squared error (MSE) under communication MSE
constraints through dedicated hybrid transceiver design. Optimization involves
the analog part through a min-MSE-based beam selection algorithm, followed by
the digital part using an alternating optimization algorithm. Convergence and
asymptotic pairwise error probability (APEP) analyses accompany numerical
simulations, validating its overall enhanced ISAC performance over existing
alternatives.
