  In a Non-Orthogonal Unicast and Multicast (NOUM) transmission system, a
multicast stream intended to all the receivers is superimposed in the power
domain on the unicast streams. One layer of Successive Interference
Cancellation (SIC) is required at each receiver to remove the multicast stream
before decoding its intended unicast stream. In this paper, we first show that
a linearly-precoded 1-layer Rate-Splitting (RS) strategy at the transmitter can
efficiently exploit this existing SIC receiver architecture. We further propose
multi-layer transmission strategies based on the generalized RS and
power-domain Non-Orthogonal Multiple Access (NOMA). Two different objectives
are studied for the design of the precoders, namely, maximizing the Weighted
Sum Rate (WSR) of the unicast messages and maximizing the system Energy
Efficiency (EE), both subject to Quality of Service (QoS) rate requirements of
all the messages and a sum power constraint. A Weighted Minimum Mean Square
Error (WMMSE)-based algorithm and a Successive Convex Approximation (SCA)-based
algorithm are proposed to solve the WSR and EE problems, respectively.
Numerical results show that the proposed RS-assisted NOUM transmission
strategies are more spectrally and energy efficient than the conventional
Multi-User Linear-Precoding (MU-LP), Orthogonal Multiple Access (OMA) and
power-domain NOMA in a wide range of user deployments (with a diversity of
channel directions, channel strengths and qualities of channel state
information at the transmitter) and network loads (underloaded and overloaded
regimes). It is superior for the downlink multi-antenna NOUM transmission.
