  Let $X_1, \ldots, X_n$ be i.i.d. sample in $\mathbb{R}^p$ with zero mean and
the covariance matrix $\mathbf{\Sigma^*}$. The classical PCA approach recovers
the projector $\mathbf{P^*_{\mathcal{J}}}$ onto the principal eigenspace of
$\mathbf{\Sigma^*}$ by its empirical counterpart
$\mathbf{\widehat{P}_{\mathcal{J}}}$. Recent paper [Koltchinskii, Lounici
(2017)] investigated the asymptotic distribution of the Frobenius distance
between the projectors $\| \mathbf{\widehat{P}_{\mathcal{J}}} -
\mathbf{P^*_{\mathcal{J}}} \|_2$, while [Naumov et al. (2017)] offered a
bootstrap procedure to measure uncertainty in recovering this subspace
$\mathbf{P^*_{\mathcal{J}}}$ even in a finite sample setup. The present paper
considers this problem from a Bayesian perspective and suggests to use the
credible sets of the pseudo-posterior distribution on the space of covariance
matrices induced by the conjugated Inverse Wishart prior as sharp confidence
sets. This yields a numerically efficient procedure. Moreover, we theoretically
justify this method and derive finite sample bounds on the corresponding
coverage probability. Contrary to [Koltchinskii, Lounici (2017), Naumov et al.
(2017)], the obtained results are valid for non-Gaussian data: the main
assumption that we impose is the concentration of the sample covariance
$\mathbf{\widehat{\Sigma}}$ in a vicinity of $\mathbf{\Sigma^*}$. Numerical
simulations illustrate good performance of the proposed procedure even on
non-Gaussian data in a rather challenging regime.
