  The demand for providing multicast services in cellular networks is
continuously and fastly increasing. In this work, we propose a non-orthogonal
transmission framework based on layered-division multiplexing (LDM) to support
multicast and unicast services concurrently in cooperative multi-cell cellular
networks with limited backhaul capacity. We adopt a two-layer LDM structure
where the first layer is intended for multicast services, the second layer is
for unicast services, and the two layers are superposed with different
beamformers. Each user decodes the multicast message first, subtracts it, and
then decodes its dedicated unicast message. We formulate a joint multicast and
unicast beamforming problem with adaptive base station clustering that aims to
maximize the weighted sum of the multicast rate and the unicast rate under
per-BS power and backhaul constraints. To solve the problem, we first develop a
branch-and-bound algorithm to find its global optimum. We then reformulate the
problem as a sparse beamforming problem and propose a low-complexity algorithm
based on convex-concave procedure. Simulation results demonstrate the
significant superiority of the proposed LDM-based non-orthogonal scheme over
orthogonal schemes in terms of the achievable multicast-unicast rate region.
