  In recent years, the multiple-input multiple-output (MIMO) non-orthogonal
multiple-access (NOMA) systems have attracted a significant interest in the
relevant research communities. As a potential precoding scheme, the generalized
singular value decomposition (GSVD) can be adopted in MIMO-NOMA systems and has
been proved to have high spectral efficiency. In this paper, the performance of
the GSVD-based MIMO-NOMA communications with Rician fading is studied. In
particular, the distribution characteristics of generalized singular values
(GSVs) of channel matrices are analyzed. Two novel mathematical tools, the
linearization trick and the deterministic equivalent method, which are based on
operator-valued free probability theory, are exploited to derive the Cauchy
transform of GSVs. An iterative process is proposed to obtain the numerical
values of the Cauchy transform of GSVs, which can be exploited to derive the
average data rates of the communication system. In addition, the special case
when the channel is modeled as Rayleigh fading, i.e., the line-of-sight
propagation is trivial, is analyzed. In this case, the closed-form expressions
of average rates are derived from the proposed iterative process. Simulation
results are provided to validate the derived analytical results.
