  In this paper, we study robust covariance estimation under the approximate
factor model with observed factors. We propose a novel framework to first
estimate the initial joint covariance matrix of the observed data and the
factors, and then use it to recover the covariance matrix of the observed data.
We prove that once the initial matrix estimator is good enough to maintain the
element-wise optimal rate, the whole procedure will generate an estimated
covariance with desired properties. For data with only bounded fourth moments,
we propose to use Huber loss minimization to give the initial joint covariance
estimation. This approach is applicable to a much wider range of distributions,
including sub-Gaussian and elliptical distributions. We also present an
asymptotic result for Huber's M-estimator with a diverging parameter. The
conclusions are demonstrated by extensive simulations and real data analysis.
