  We show that in a common high-dimensional covariance model, the choice of
loss function has a profound effect on optimal estimation. In an asymptotic
framework based on the Spiked Covariance model and use of orthogonally
invariant estimators, we show that optimal estimation of the population
covariance matrix boils down to design of an optimal shrinker $\eta$ that acts
elementwise on the sample eigenvalues. Indeed, to each loss function there
corresponds a unique admissible eigenvalue shrinker $\eta^*$ dominating all
other shrinkers. The shape of the optimal shrinker is determined by the choice
of loss function and, crucially, by inconsistency of both eigenvalues and
eigenvectors of the sample covariance matrix. Details of these phenomena and
closed form formulas for the optimal eigenvalue shrinkers are worked out for a
menagerie of 26 loss functions for covariance estimation found in the
literature, including the Stein, Entropy, Divergence, Frechet,
Bhattacharya/Matusita, Frobenius Norm, Operator Norm, Nuclear Norm and
Condition Number losses.
