  We address receiver (RX) signal processing in MIMO systems under
in-phase/quadrature (I/Q) imbalance, which causes cross-talk of
mirror-subcarriers in OFDM systems. We extend the typically reported
single-user studies to uplink OFDMA-based multiuser MIMO, with simultaneous
user multiplexing in frequency and spatial domains. We also incorporate
multiple external interferers, for modeling challenging conditions in
heterogeneous networks. In the signal processing developments, we exploit the
augmented subcarrier processing, which processes each subcarrier jointly with
its counterpart at the image subcarrier, and jointly across RX antennas.
Furthermore, we derive an augmented LMMSE RX. The novel approach integrates the
I/Q imbalance mitigation, interference suppression and data stream separation
into a single processing stage, thus avoiding a separate transceiver
calibration. Our numerical results show the signal-to-interference-plus-noise
ratio and symbol-error rate of an arbitrary data stream after RX processing as
a function of different system parameters. The per-subcarrier processing is
shown to suffer heavily under I/Q imbalances, and being particularly sensitive
to external interferers, whereas the augmented method provides efficient data
stream separation and interference suppression. Finally, we extend the studies
to massive MIMO framework and show that the per-subcarrier processing still
suffers from performance degradation, whereas the augmented approach can fully
exploit the array gain.
