  Principal component analysis (PCA) is a standard tool for dimensional
reduction of a set of $n$ observations (samples), each with $p$ variables. In
this paper, using a matrix perturbation approach, we study the nonasymptotic
relation between the eigenvalues and eigenvectors of PCA computed on a finite
sample of size $n$, and those of the limiting population PCA as $n\to\infty$.
As in machine learning, we present a finite sample theorem which holds with
high probability for the closeness between the leading eigenvalue and
eigenvector of sample PCA and population PCA under a spiked covariance model.
In addition, we also consider the relation between finite sample PCA and the
asymptotic results in the joint limit $p,n\to\infty$, with $p/n=c$. We present
a matrix perturbation view of the "phase transition phenomenon," and a simple
linear-algebra based derivation of the eigenvalue and eigenvector overlap in
this asymptotic limit. Moreover, our analysis also applies for finite $p,n$
where we show that although there is no sharp phase transition as in the
infinite case, either as a function of noise level or as a function of sample
size $n$, the eigenvector of sample PCA may exhibit a sharp "loss of tracking,"
suddenly losing its relation to the (true) eigenvector of the population PCA
matrix. This occurs due to a crossover between the eigenvalue due to the signal
and the largest eigenvalue due to noise, whose eigenvector points in a random
direction.
