  We propose a Bayesian methodology for estimating spiked covariance matrices
with jointly sparse structure in high dimensions. The spiked covariance matrix
is reparametrized in terms of the latent factor model, where the loading matrix
is equipped with a novel matrix spike-and-slab LASSO prior, which is a
continuous shrinkage prior for modeling jointly sparse matrices. We establish
the rate-optimal posterior contraction for the covariance matrix with respect
to the operator norm as well as that for the principal subspace with respect to
the projection operator norm loss. We also study the posterior contraction rate
of the principal subspace with respect to the two-to-infinity norm loss, a
novel loss function measuring the distance between subspaces that is able to
capture element-wise eigenvector perturbations. We show that the posterior
contraction rate with respect to the two-to-infinity norm loss is tighter than
that with respect to the routinely used projection operator norm loss under
certain low-rank and bounded coherence conditions. In addition, a point
estimator for the principal subspace is proposed with the rate-optimal risk
bound with respect to the projection operator norm loss. These results are
based on a collection of concentration and large deviation inequalities for the
matrix spike-and-slab LASSO prior. The numerical performance of the proposed
methodology is assessed through synthetic examples and the analysis of a
real-world face data example.
