  We propose a joint channel estimation and signal detection approach for the
uplink non-orthogonal multiple access (NOMA) using unsupervised machine
learning. We apply a Gaussian mixture model (GMM) to cluster the received
signals, and accordingly optimize the decision regions to enhance the symbol
error rate (SER) performance. We show that, when the received powers of the
users are sufficiently different, the proposed clustering-based approach
achieves an SER performance on a par with that of the conventional
maximum-likelihood detector (MLD) with full channel state information (CSI). We
study the tradeoff between the accuracy of the proposed approach and the
blocklength, as the accuracy of the utilized clustering algorithm depends on
the number of symbols available at the receiver. We provide a comprehensive
performance analysis of the proposed approach and derive a theoretical bound on
its SER performance. Our simulation results corroborate the effectiveness of
the proposed approach and verify that the calculated theoretical bound can
predict the SER performance of the proposed approach well. We further explore
the application of the proposed approach to a practical grant-free NOMA
scenario, and show that its performance is very close to that of the optimal
MLD with full CSI, which usually requires long pilot sequences.
