  We study the supervised clustering problem under the two-component
anisotropic Gaussian mixture model in high dimensions and in the non-asymptotic
setting. We first derive a lower and a matching upper bound for the minimax
risk of clustering in this framework. We also show that in the high-dimensional
regime, the linear discriminant analysis (LDA) classifier turns out to be
sub-optimal in the minimax sense. Next, we characterize precisely the risk of
$\ell_2$-regularized supervised least squares classifiers. We deduce the fact
that the interpolating solution may outperform the regularized classifier,
under mild assumptions on the covariance structure of the noise. Our analysis
also shows that interpolation can be robust to corruption in the covariance of
the noise when the signal is aligned with the "clean" part of the covariance,
for the properly defined notion of alignment. To the best of our knowledge,
this peculiar phenomenon has not yet been investigated in the rapidly growing
literature related to interpolation. We conclude that interpolation is not only
benign but can also be optimal, and in some cases robust.
