  Estimating the number of signals embedded in noise is a fundamental problem
in array signal processing. The classic RMT estimator based on random matrix
theory (RMT) tends to under-estimate the number of signals as it does not
consider the non-negligible bias term among eigenvalues for finite sample size.
Moreover, the RMT estimator suffers from uncertainty in noise variance
estimation problem. In order to overcome these problems, we firstly derive a
more accurate expression for the distribution of the sample eigenvalues and the
bias term among eigenvalues by utilizing the linear shrinkage (LS) estimate of
noise sample eigenvalues. Then, we analyze the effect of the bias term among
eigenvalues on the estimation performance of the RMT estimator, and derive the
increased under-estimation probability of the RMT estimator incurred by this
bias term. Based on these results, we propose a novel RMT estimator based on LS
estimate of noise eigenvalues (termed as LS-RMT estimator) by incorporating the
bias term into the decision criterion of the RMT estimator. As the LS-RMT
estimator incorporates this bias term among eigenvalues into the decision
criterion of the RMT estimator, it can detect signal eigenvalues immersed in
this bias term. Therefore, the LS-RMT estimator can overcome the higher
under-estimation probability of the RMT estimator incurred by the bias term
among eigenvalues, and also avoids the uncertainty in the noise variance
estimation suffered by the RMT estimator as the noise variance is estimated
under the assumption that the eigenvalue being tested is arising from noise.
Finally, extensive simulation results are presented to show that the proposed
LS-RMT estimator outperforms the existing estimators.
