  We develop a data-driven optimal shrinkage algorithm for matrix denoising in
the presence of high-dimensional noise with a separable covariance structure;
that is, the noise is colored and dependent across samples. The algorithm,
coined {\em extended OptShrink} (eOptShrink) depends on the asymptotic behavior
of singular values and singular vectors of the random matrix associated with
the noisy data. Based on the developed theory, including the sticking property
of non-outlier singular values and delocalization of the non-outlier singular
vectors associated with weak signals with a convergence rate, and the spectral
behavior of outlier singular values and vectors, we develop three estimators,
each of these has its own interest. First, we design a novel rank estimator,
based on which we provide an estimator for the spectral distribution of the
pure noise matrix, and hence the optimal shrinker called eOptShrink. In this
algorithm we do not need to estimate the separable covariance structure of the
noise. A theoretical guarantee of these estimators with a convergence rate is
given. On the application side, in addition to a series of numerical
simulations with a comparison with various state-of-the-art optimal shrinkage
algorithms, we apply eOptShrink to extract maternal and fetal
electrocardiograms from the single channel trans-abdominal maternal
electrocardiogram.
