  The integrated sensing and communication (ISAC) system under multi-input
multi-output (MIMO) architecture achieves dual functionalities of sensing and
communication on the same platform by utilizing spatial gain, which provides a
feasible paradigm facing spectrum congestion. However, the dual functionalities
of sensing and communication operating simultaneously in the same platform
bring severe interference in the ISAC systems. Facing this challenge, we
propose a joint optimization framework for transmit beamforming and receive
filter design for ISAC systems with MIMO architecture. We aim to maximize the
signal-to-clutter-plus-noise ratio (SCNR) at the receiver while considering
various constraints such as waveform similarity, power budget, and
communication performance requirements to ensure the integration of the dual
functionalities. In particular, the overall transmit beamforming is refined
into sensing beamforming and communication beamforming, and a quadratic
transformation (QT) is introduced to relax and convert the complex non-convex
optimization objective. An efficient algorithm based on covariance matrix
tapers (CMT) is proposed to restructure the clutter covariance matrix
considering the mismatched steering vector, thereby improving the robustness of
the ISAC transceiver design. Numerical simulations are provided to demonstrate
the effectiveness of the proposed algorithm.
