  Low-complexity near-optimal detection of large-MIMO signals has attracted
recent research. Recently, we proposed a local neighborhood search algorithm,
namely `reactive tabu search' (RTS) algorithm, as well as a factor-graph based
`belief propagation' (BP) algorithm for low-complexity large-MIMO detection.
The motivation for the present work arises from the following two observations
on the above two algorithms: $i)$ RTS works for general M-QAM. Although RTS was
shown to achieve close to optimal performance for 4-QAM in large dimensions,
significant performance improvement was still possible for higher-order QAM
(e.g., 16- and 64-QAM). ii) BP also was shown to achieve near-optimal
performance for large dimensions, but only for $\{\pm 1\}$ alphabet. In this
paper, we improve the large-MIMO detection performance of higher-order QAM
signals by using a hybrid algorithm that employs RTS and BP. In particular,
motivated by the observation that when a detection error occurs at the RTS
output, the least significant bits (LSB) of the symbols are mostly in error, we
propose to first reconstruct and cancel the interference due to bits other than
LSBs at the RTS output and feed the interference cancelled received signal to
the BP algorithm to improve the reliability of the LSBs. The output of the BP
is then fed back to RTS for the next iteration. Our simulation results show
that in a 32 x 32 V-BLAST system, the proposed RTS-BP algorithm performs better
than RTS by about 3.5 dB at $10^{-3}$ uncoded BER and by about 2.5 dB at
$3\times 10^{-4}$ rate-3/4 turbo coded BER with 64-QAM at the same order of
complexity as RTS. We also illustrate the performance of large-MIMO detection
in frequency-selective fading channels.
