  In this paper, we investigate covert communications in a simultaneously
transmitting and reflecting reconfigurable intelligent surface (STAR-RIS)-aided
rate-splitting multiple access (RSMA) system. Under the RSMA principles, the
messages for the covert user (Bob) and public user (Grace) are converted to the
common and private streams at the legitimate transmitter (Alice) to realize
downlink transmissions, while the STAR-RIS is deployed not only to aid the
public transmissions from Alice to Grace, but also to shield the covert
transmissions from Alice to Bob against the warden (Willie). To characterize
the covert performance of the considered STAR-RIS-aided RSMA (STAR-RIS-RSMA)
system, we derive analytical expression for the minimum average detection error
probability of Willie, based on which a covert rate maximization problem is
formulated. To maximize Bob's covert rate while confusing Willie's monitoring,
the transmit power allocation, common rate allocation, and STAR-RIS
reflection/transmission beamforming are jointly optimized subject to Grace's
quality of service (QoS) requirements. The non-convex covert rate maximization
problem, consisting of highly coupled system parameters are decoupled into
three sub-problems of transmit power allocation, common rate allocation, and
STAR-RIS reflection/transmission beamforming, respectively. To obtain the
rank-one constrained optimal solution for the sub-problem of optimizing the
STAR-RIS reflection/transmission beamforming, a penalty-based successive convex
approximation scheme is developed. Moreover, an alternative optimization (AO)
algorithm is designed to determine the optimal solution for the sub-problem of
optimizing the transmit power allocation, while the original problem is overall
solved by a new AO algorithm.
