  In this paper, a general class of regularized $M$-estimators of scatter
matrix are proposed which are suitable also for low or insufficient sample
support (small $n$ and large $p$) problems. The considered class constitutes a
natural generalization of $M$-estimators of scatter matrix (Maronna, 1976) and
are defined as a solution to a penalized $M$-estimation cost function that
depend on a pair $(\alpha,\beta)$ of regularization parameters. We derive
general conditions for uniqueness of the solution using concept of geodesic
convexity. Since these conditions do not include Tyler's $M$-estimator,
necessary and sufficient conditions for uniqueness of the penalized Tyler's
cost function are established separately. For the regularized Tyler's
$M$-estimator, we also derive a simple, closed form and data dependent solution
for choosing the regularization parameter based on shape matrix matching in the
mean squared sense. An iterative algorithm that converges to the solution of
the regularized $M$-estimating equation is also provided. Finally, some
simulations studies illustrate the improved accuracy of the proposed
regularized $M$-estimators of scatter compared to their non-regularized
counterparts in low sample support problems. An example of radar detection
using normalized matched filter (NMF) illustrate that an adaptive NMF detector
based on regularized $M$-estimators are able to maintain accurately the preset
CFAR level and at at the same time provide similar probability of detection as
the (theoretical) NMF detector.
