  This study considers a novel full-duplex (FD) massive multiple-input
multiple-output (mMIMO) system using hybrid beamforming (HBF) architecture,
which allows for simultaneous uplink (UL) and downlink (DL) transmission over
the same frequency band. Particularly, our objective is to mitigate the strong
self-interference (SI) solely on the design of UL and DL RF beamforming stages
jointly with sub-array selection (SAS) for transmit (Tx) and receive (Rx)
sub-arrays at base station (BS). Based on the measured SI channel in an
anechoic chamber, we propose a min-SI beamforming scheme with SAS, which
applies perturbations to the beam directivity to enhance SI suppression in UL
and DL beam directions. To solve this challenging nonconvex optimization
problem, we propose a swarm intelligence-based algorithmic solution to find the
optimal perturbations as well as the Tx and Rx sub-arrays to minimize SI
subject to the directivity degradation constraints for the UL and DL beams. The
results show that the proposed min-SI BF scheme can achieve SI suppression as
high as 78 dB in FD mMIMO systems.
