  This article addresses improvements on the design of the adaptive normalized
matched filter (ANMF) for radar detection. It is well-acknowledged that the
estimation of the noise-clutter covariance matrix is a fundamental step in
adaptive radar detection. In this paper, we consider regularized estimation
methods which force by construction the eigenvalues of the scatter estimates to
be greater than a positive regularization parameter rho. This makes them more
suitable for high dimensional problems with a limited number of secondary data
samples than traditional sample covariance estimates. While an increase of rho
seems to improve the conditioning of the estimate, it might however cause it to
significantly deviate from the true covariance matrix. The setting of the
optimal regularization parameter is a difficult question for which no
convincing answers have thus far been provided. This constitutes the major
motivation behind our work. More specifically, we consider the design of the
ANMF detector for two kinds of regularized estimators, namely the regularized
sample covariance matrix (RSCM), appropriate when the clutter follows a
Gaussian distribution and the regularized Tyler estimator (RTE) for
non-Gaussian spherically invariant distributed clutters. Based on recent random
matrix theory results studying the asymptotic fluctuations of the statistics of
the ANMF detector when the number of samples and their dimension grow together
to infinity, we propose a design for the regularization parameter that
maximizes the detection probability under constant false alarm rates.
Simulation results which support the efficiency of the proposed method are
provided in order to illustrate the gain of the proposed optimal design over
conventional settings of the regularization parameter.
