  Non-orthogonal multiple access (NOMA) is widely recognized as a potential
multiple access technology for efficient radio spectrum utilization in the
fifth-generation (5G) wireless communications standard. In this paper, we study
the average achievable rate and outage probability of a cooperative relaying
system (CRS) based on NOMA (CRS-NOMA) over wireless links governed by the
$\alpha$-$\mu$ generalized fading model; here $\alpha$ and $\mu$ designate the
nonlinearity and clustering parameters, respectively, of each link. The average
achievable rate is represented in closed-form using Meijer's G-function and the
extended generalized bivariate Fox's H-function (EGBFHF), and the outage
probability is represented using the lower incomplete Gamma function. Our
results confirm that the CRS-NOMA outperforms the CRS with conventional
orthogonal multiple access (CRS-OMA) in terms of spectral efficiency at high
transmit signal-to-noise ratio (SNR). It is also evident from our results that
with an increase in the value of the nonlinearity/clustering parameter, the SNR
at which the CRS-NOMA outperforms its OMA based counterpart becomes higher.
Furthermore, the asymptotic analysis of the outage probability reveals the
dependency of the diversity order of each symbol in the CRS-NOMA system on the
$\alpha$ and $\mu$ parameters of the fading links.
