  In the practical radar with multiple antennas, the antenna imperfections
degrade the system performance. In this paper, the problem of estimating the
direction of arrival (DOA) in multiple-input and multiple-output (MIMO) radar
system with unknown mutual coupling effect between antennas is investigated. To
exploit the target sparsity in the spatial domain, the compressed sensing
(CS)-based methods have been proposed by discretizing the detection area and
formulating the dictionary matrix, so an \emph{off-grid} gap is caused by the
discretization processes. In this paper, different from the present DOA
estimation methods, both the off-grid gap due to the sparse sampling and the
unknown mutual coupling effect between antennas are considered at the same
time, and a novel sparse system model for DOA estimation is formulated. Then, a
novel sparse Bayesian learning (SBL)-based method named sparse Bayesian
learning with the mutual coupling (SBLMC) is proposed, where an
expectation-maximum (EM)-based method is established to estimate all the
unknown parameters including the noise variance, the mutual coupling vectors,
the off-grid vector and the variance vector of scattering coefficients.
Additionally, the prior distributions for all the unknown parameters are
theoretically derived. With regard to the DOA estimation performance, the
proposed SBLMC method can outperform state-of-the-art methods in the MIMO radar
with unknown mutual coupling effect, while keeping the acceptable computational
complexity.
