  In this paper, we exploit the spiked covariance structure of the clutter plus
noise covariance matrix for radar signal processing. Using state-of-the-art
techniques high dimensional statistics, we propose a nonlinear shrinkage-based
rotation invariant spiked covariance matrix estimator. We state the convergence
of the estimated spiked eigenvalues. We use a dataset generated from the
high-fidelity, site-specific physics-based radar simulation software RFView to
compare the proposed algorithm against the existing Rank Constrained Maximum
Likelihood (RCML)-Expected Likelihood (EL) covariance estimation algorithm. We
demonstrate that the computation time for the estimation by the proposed
algorithm is less than the RCML-EL algorithm with identical Signal to Clutter
plus Noise (SCNR) performance. We show that the proposed algorithm and the
RCML-EL-based algorithm share the same optimization problem in high dimensions.
We use Low-Rank Adaptive Normalized Matched Filter (LR-ANMF) detector to
compute the detection probabilities for different false alarm probabilities
over a range of target SNR. We present preliminary results which demonstrate
the robustness of the detector against contaminating clutter discretes using
the Challenge Dataset from RFView. Finally, we empirically show that the
minimum variance distortionless beamformer (MVDR) error variance for the
proposed algorithm is identical to the error variance resulting from the true
covariance matrix.
