  In this paper, we investigate signal detection in
multiple-input-multiple-output (MIMO) communication systems with hardware
impairments, such as power amplifier nonlinearity and in-phase/quadrature
imbalance. To deal with the complex combined effects of hardware imperfections,
neural network (NN) techniques, in particular deep neural networks (DNNs), have
been studied to directly compensate for the impact of hardware impairments.
However, it is difficult to train a DNN with limited pilot signals, hindering
its practical applications. In this work, we investigate how to achieve
efficient Bayesian signal detection in MIMO systems with hardware
imperfections. Characterizing combined hardware imperfections often leads to
complicated signal models, making Bayesian signal detection challenging. To
address this issue, we first train an NN to "model" the MIMO system with
hardware imperfections and then perform Bayesian inference based on the trained
NN. Modelling the MIMO system with NN enables the design of NN architectures
based on the signal flow of the MIMO system, minimizing the number of NN layers
and parameters, which is crucial to achieving efficient training with limited
pilot signals. We then represent the trained NN with a factor graph, and design
an efficient message passing based Bayesian signal detector, leveraging the
unitary approximate message passing (UAMP) algorithm. The implementation of a
turbo receiver with the proposed Bayesian detector is also investigated.
Extensive simulation results demonstrate that the proposed technique delivers
remarkably better performance than state-of-the-art methods.
