  This work focuses on the use of reconfigurable intelligent surface (RIS) in
dual-function radar-communication (DFRC) systems to improve communication
capacity and sensing precision, and enhance coverage for both functions. In
contrast to most of the existing RIS aided DFRC works where the RIS is modeled
as a diagonal phase shift matrix and can only reflect signals to half space, we
propose a novel beyond diagonal RIS (BD-RIS) aided DFRC system. Specifically,
the proposed BD-RIS supports the hybrid reflecting and transmitting mode, and
is compatible with flexible architectures, enabling the system to realize
full-space coverage and to achieve enhanced performance. To achieve the
expected benefits, we jointly optimize the transmit waveform, the BD-RIS
matrices, and sensing receive filters, by maximizing the minimum
signal-to-clutter-plus-noise ratio for fair target detection, subject to the
constraints of the communication quality of service, different BD-RIS
architectures and power budget. To solve the non-convex and non-smooth max-min
problem, a general solution based on the alternating direction method of
multipliers is provided. Numerical simulations validate the efficacy of the
proposed algorithm and show the superiority of the BD-RIS aided DFRC system in
terms of both communication and sensing compared to conventional RIS aided
DFRC.
