  A novel coexisting passive reconfigurable intelligent surface (RIS) and
active decode-and-forward (DF) relay assisted non-orthogonal multiple access
(NOMA) transmission framework is proposed. In particular, two communication
protocols are conceived, namely Hybrid NOMA (H-NOMA) and Full NOMA (F-NOMA).
Based on the proposed two protocols, both the sum rate maximization and max-min
rate fairness problems are formulated for jointly optimizing the power
allocation at the access point and relay as well as the passive beamforming
design at the RIS. To tackle the non-convex problems, an alternating
optimization (AO) based algorithm is first developed, where the transmit power
and the RIS phase-shift are alternatingly optimized by leveraging the
two-dimensional search and rank-relaxed difference-of-convex (DC) programming,
respectively. Then, a two-layer penalty based joint optimization (JO) algorithm
is developed to jointly optimize the resource allocation coefficients within
each iteration. Finally, numerical results demonstrate that: i) the proposed
coexisting RIS and relay assisted transmission framework is capable of
achieving a significant user performance improvement than conventional schemes
without RIS or relay; ii) compared with the AO algorithm, the JO algorithm
requires less execution time at the cost of a slight performance loss; and iii)
the H-NOMA and F-NOMA protocols are generally preferable for ensuring user rate
fairness and enhancing user sum rate, respectively.
