  Integrated sensing and communication (ISAC) has been regarded as one of the
most promising technologies for future wireless communications. However, the
mutual interference in the communication radar coexistence system cannot be
ignored. Inspired by the studies of reconfigurable intelligent surface (RIS),
we propose a double-RIS-assisted coexistence system where two RISs are deployed
for enhancing communication signals and suppressing mutual interference. We aim
to jointly optimize the beamforming of RISs and radar to maximize communication
performance while maintaining radar detection performance. The investigated
problem is challenging, and thus we transform it into an equivalent but more
tractable form by introducing auxiliary variables. Then, we propose a penalty
dual decomposition (PDD)-based algorithm to solve the resultant problem.
Moreover, we consider two special cases: the large radar transmit power
scenario and the low radar transmit power scenario. For the former, we prove
that the beamforming design is only determined by the communication channel and
the corresponding optimal joint beamforming strategy can be obtained in
closed-form. For the latter, we minimize the mutual interference via the block
coordinate descent (BCD) method. By combining the solutions of these two cases,
a low-complexity algorithm is also developed. Finally, simulation results show
that both the PDD-based and low-complexity algorithms outperform benchmark
algorithms.
