  This study employs a uniform rectangular array (URA) sub-connected hybrid
beamforming (SC-HBF) architecture to provide a novel self-interference (SI)
suppression scheme in a full-duplex (FD) massive multiple-input multiple-output
(mMIMO) system. Our primary objective is to mitigate the strong SI through the
design of RF beamforming stages for uplink and downlink transmissions that
utilize the spatial degrees of freedom provided due to the use of large array
structures. We propose a non-constant modulus RF beamforming (NCM-BF-SIS)
scheme that incorporates the gain controllers for both transmit (Tx) and
receive (Rx) RF beamforming stages and optimizes the uplink and downlink beam
directions jointly with gain controller coefficients. To solve this challenging
non-convex optimization problem, we propose a swarm intelligence-based
algorithmic solution that finds the optimal beam perturbations while also
adjusting the Tx/Rx gain controllers to alleviate SI subject to the directivity
degradation constraints for the beams. The data-driven analysis based on the
measured SI channel in an anechoic chamber shows that the proposed NCM-BF-SIS
scheme can suppress SI by around 80 dB in FD mMIMO systems.
