  This paper investigates the sum spectral efficiency maximization problem in
downlink multiuser multiple-input multiple-output (MIMO) systems with
low-resolution quantizers at an access point (AP) and users. In particular, we
consider rate-splitting multiple access (RSMA) to enhance spectral efficiency
by offering opportunities to boost achievable degrees of freedom. Optimizing
RSMA precoders, however, is highly challenging due to the minimum rate
constraint when determining the rate of the common stream. The quantization
errors coupled with the precoders further make the problem more complicated and
difficult to solve. In this paper, we develop a novel RSMA precoding algorithm
incorporating quantization errors for maximizing the sum spectral efficiency.
To this end, we first obtain an approximate spectral efficiency in a smooth
function. Subsequently, we derive the first-order optimality condition in the
form of the nonlinear eigenvalue problem (NEP). We propose a computationally
efficient algorithm to find the principal eigenvector of the NEP as a
sub-optimal solution. Simulation results validate the superior spectral
efficiency of the proposed method. The key benefit of using RSMA over spatial
division multiple access (SDMA) comes from the ability of the common stream to
balance between the channel gain and quantization error in multiuser MIMO
systems with different quantization resolutions.
