  This paper investigates the optimization of reconfigurable intelligent
surface (RIS) in an integrated sensing and communication (ISAC) system. \red{To
meet the demand of growing number of devices, power domain non-orthogonal
multiple access (NOMA) is considered. However, traditional NOMA with a large
number of devices is challenging due to large decoding delay and propagation
error introduced by successive interference cancellation (SIC). Thus, OMA is
integrated into NOMA to support more devices}. We formulate a max-min problem
to optimize the sensing beampattern \red{with constraints on communication
rate}, through joint power allocation, active beamforming and RIS phase shift
design. To solve the non-convex problem with a non-smooth objective function,
we propose a low complexity alternating optimization (AO) algorithm, where a
closed form expression for the intra-cluster power allocation (intra-CPA) is
derived, and penalty and successive convex approximation (SCA) methods are used
to optimize the beamforming and phase shift design. Simulation results show the
effectiveness of the proposed algorithm in terms of improving minimum
beampattern gain (MBPG) compared with other baselines. Furthermore, the
trade-off between sensing and communication is analyzed and demonstrated in the
simulation results.
