  In this paper, we propose a non-binary belief propagation approach (NB-BP)
for detection of $M$-ary modulation symbols and decoding of $q$-ary LDPC codes
in large-scale multiuser MIMO systems. We first propose a message passing based
symbol detection algorithm which computes vector messages using a scalar
Gaussian approximation of interference, which results in a total complexity of
just $O(KN\sqrt{M})$, where $K$ is the number of uplink users and $N$ is the
number of base station (BS) antennas. The proposed NB-BP detector does not need
to do a matrix inversion, which gives a complexity advantage over MMSE
detection. We then design optimized $q$-ary LDPC codes by matching the EXIT
charts of the proposed detector and the LDPC decoder. Simulation results show
that the proposed NB-BP detection-decoding approach using the optimized LDPC
codes achieve significantly better performance (by about 1 dB to 7 dB at
$10^{-5}$ coded BER for various system loading factors with number of users
ranging from 16 to 128 and number of BS antennas fixed at 128) compared to
using linear detectors (e.g., MMSE detector) and off-the-shelf $q$-ary
irregular LDPC codes. Also, even with estimated channel knowledge (e.g., with
MMSE channel estimate), the performance of the proposed NB-BP detector is
better than that of the MMSE detector.
