Cross-Validation for Unsupervised Learning
Patrick O. Perry
abstract.tex
Cross-validation (CV) is a popular method for model-selection.  Unfortunately, it is not immediately obvious how to apply CV to unsupervised or exploratory contexts.  This thesis discusses some extensions of cross-validation to unsupervised learning, specifically focusing on the problem of choosing how many principal components to keep.  We introduce the latent factor model, define an objective criterion, and show how CV can be used to estimate the intrinsic dimensionality of a data set.  Through both simulation and theory, we demonstrate that cross-validation is a valuable tool for unsupervised learning.


acknowledgment.tex
\prefacesection{Acknowledgments}

This work could not have been done alone.  I would like to thank:
\begin{itemize}
    \item Art Owen for always having an open door and for providing
        endless advice and encouragement;
    \item Gunnar Carlsson, Trevor Hastie, Iain Johnstone, 
        and Jonathan Taylor for valuable feedback;
    \item Mollie Biewald, Ray, Kerry, Clare, and Erin Perry for
        unwavering support;
    \item the entire Stanford Statistics Department for community and
        education.
\end{itemize}
I would also like to thank Mark Churchland for providing the motor
cortex data.  This work was supported by a Stanford Graduate Fellowship
and by the National Science Foundation under Grant DMS-0652743.

bcv-theory.tex
\chapter[A theoretical analysis of BCV]{A theoretical analysis of bi-cross-validation}
\label{C:optimal-leave-out}

In this chapter we will determine the optimal leave out-size for Gabriel-style
cross-validation of an SVD, also known as bi-cross-validation (BCV), along
with proving a weak form of consistency.  In Chapter~\ref{C:intrinsic-rank},
we rigorously defined the rank estimation problem, and in 
Chapter~\ref{C:cvul} we introduced Gabriel-style cross validation.  Here, we 
provide theoretic justification for Gabriel-style CV.

First, a quick review of the problem.  We are given $\mX$, an $n \times p$ matrix generating by a ``signal-plus-noise'' process, 
\[
    \mX = \sqrt{n} \, \mU \mD \mV^\trans + \mE.
\]
Here, $\mU \in \reals^{n \times k_0}$, $\mD \in \reals^{k_0 \times k_0}$,
$\mV \in \reals^{p \times k_0}$, and $\mE \in \reals^{n \times p}$.  The first
term, $\sqrt{n} \, \mU \mD \mV^\trans$, is the low-rank ``signal'' part.  We call 
$\mU$ and $\mV$ the matrices of left and right factors, respectively.  They 
are normalized so that $\mU^\trans \mU = \mV^\trans \mV = \mI_{k_0}$.  The 
factor ``strengths'' are given in $\mD$, a diagonal matrix of the form
\(
    \mD =
        \diag( d_1, d_2, \ldots, d_{k_0}),
\)
with $d_1 \geq d_2 \geq \cdots \geq d_{k_0} \geq 0$.  Also, typically $k_0$ is 
much smaller than $n$ and $p$.  Lastly, $\mE$ consists of ``noise''.  Although more general types of noise are possible, for simplicity we will assume that $\mE$ is independent of $\mU$, $\mD$, and $\mV$.  We think 
of the signal part as the important part of $\mX$, and the noise part is 
inherently uninteresting.

The rank estimation problem is find the optimal number of terms of the SVD to keep to estimate the signal part.  We let 
$\mX = \sqrt{n} \, \mhU \mhD \mhV^\trans$ be the SVD of $\mX$, where 
$\mhU \in \reals^{n \times n \wedge p}$ and 
$\mhV \in \reals^{p \times n \wedge p}$ have orthonormal columns, and
\(
    \mhD = \diag(
        \hd_1, \hd_2, \ldots, \hd_{n \wedge p}
    )
\)
with $\hd_1 \geq \hd_2 \geq \cdots \geq \hd_{n \wedge p}$.  For $0 \leq k \leq n \wedge p$, we define
\(
    \mhD(k) = \diag(
        \hd_1, \hd_2, \ldots, \hd_k, 0, 0, \ldots, 0
    )
\)
so that $\mhX(k) \equiv \sqrt{n} \, \mhU \mhD(k) \mhV^\trans$ is the SVD of $\mX$ truncated to $k$ terms.  The model error with respect to Frobenius loss is given by
\[
    \ME(k) = \frac{1}{n \, p}
             \| \sqrt{n} \, \mU \mD \mV^\trans - \mhX(k) \|_\Frob^2
\]
The optimal rank is defined with respect to this criterion is
\[
    k^\ast = \argmin_k \ME(k).
\]
The problem we consider is how to estimate $\ME(k)$ or $k^\ast$.

Closely related to model error is the \emph{prediction error}.  For prediction error, we conjure up a noise matrix $\mE'$ with the same distribution as $\mE$ and let $\mX' = \sqrt{n} \mU \mD \mV^\trans + \mE'$.  The prediction error is defined as
\[
    \PE(k) \equiv \frac{1}{n \, p} \E\| \mX' - \mhX(k) \|_\Frob^2,
\]
which can be expressed as
\[
    \PE(k) = \E[ \ME(k) ] + \frac{1}{n \, p} \E \| \mE \|_\Frob^2.
\]
The minimizer of $\PE$ is the same as the minimizer of $\E[ \ME(k) ]$,
and one can get an estimate of $\ME$ from an estimate of $\PE$ by subtracting an estimate of the noise level.

The previous chapter suggests using Gabriel-style cross-validation for estimating the optimal rank.  Owen \& Perry~\cite{owen2009bi} call this procedure bi-cross-validation (BCV).  For fold $(i,j)$ of BCV, we permute
the rows of $\mX$ with matrices $\mP^{(i)}$ and $\mQ^{(j)}$, then partition the result into four blocks as
\[
    \mP^{(i)\trans} \mX \mQ^{(j)}
        =
        \left(
        \begin{matrix}
            \mX_{11} & \mX_{12} \\
            \mX_{21} & \mX_{22}
        \end{matrix}
        \right).
\]
We take the SVD of the upper-left block and evaluate its predictive 
performance on the lower-right block.  If 
$\mX_{11} = \sqrt{n} \, \mhU_1 \mhD_1 \mhV^\trans$ is the SVD of $\mX_{11}$ and
$\mhX_{11}(k) = \sqrt{n} \, \mhU_1 \mhD_1(k) \mhV^\trans$ is its truncation to 
$k$ terms (with $\mhD_1(k)$ defined analogously to $\mhD(k)$), then the BCV estimate of prediction error from this fold is given 
by
\[
    \widehat{\PE}\big(k ; i, j \big)
        =
            \frac{1}{n_2 \, p_2}
            \| \mX_{22} - \mX_{21} \mhX_{11}(k)^+ \mX_{12} \|_\Frob^2.
\]
Here, $^+$ denotes pseudo-inverse and $\mX_{22}$ has dimensions $n_2 \times p_2$.  For $(K,L)$-fold BCV, the final estimate is the average over all folds:
\[
    \widehat{\PE}(k)
        =
        \frac{1}{K L}
        \sum_{i=1}^K
        \sum_{j=1}^L
            \widehat{\PE}\big(k ; i, j \big).
\]
From $\widehat{\PE}(k)$ we can get an estimate of the optimal rank as
$\hat k = \argmin_k \widehat{\PE}(k)$.

In this chapter, we give a theoretical analysis of $\widehat{\PE}(k)$. This
allows us to determine the bias inherent in $\widehat{\PE}(k)$ and its
consistency properties for estimating $k^\ast$, along with guidance for
choosing the number of folds ($K$ and $L$).
Section~\ref{S:bcv-theory-assumptions} sets out our assumptions and notation.
Section~\ref{S:bcv-theory-results} gives our main results.  Then, Sections~\ref{S:bcv-theory-holdin}~and~\ref{S:bcv-pe-estimate} are devoted to proofs, followed by a discussion in Section~\ref{S:bcv-theory-summary}.


\section{Assumptions and notation}\label{S:bcv-theory-assumptions}

The theory becomes easier if we work in an asymptotic framework.  For that,
we introduce a sequence of data matrices indexed by $n$:
\begin{equation}
    \mX_{n} = \sqrt{n} \, \mU_n \mD_n \mV_n^\trans + \mE_n.
\end{equation}
Here, $\mX_n \in \reals^{n \times p}$ with $p = p(n)$ and 
$\frac{n}{p} \to \gamma \in (0,\infty)$.  Even though the dimensions of $\mX_n$ grow, we assume that the number of factors is fixed at $k_0$.  The first set of assumptions is as follows:

\begin{assumption}\label{A:optimal-leave-out-shapes}
    We have a sequence of random matrices $\mX_n \in \reals^{n \times p}$ 
    with $n \to \infty$ and $p = p(n)$ also going to infinity.  Their ratio
    converges to a fixed constant $\gamma \in (0, \infty)$ as 
    $\frac{n}{p} = \gamma + \oh\left( \frac{1}{\sqrt{n}} \right)$.
\end{assumption}

\begin{assumption}
    The matrix $\mX_n$ is generated as 
    $\mX_n = \sqrt{n} \, \mU_n \mD_n \mV_n^\trans + \mE_n$.  Here,
    $\mU_n \in \reals^{n \times k_0}$, 
    $\mD_n \in \reals^{k_0 \times k_0}$,
    $\mV_n \in \reals^{p \times k_0}$, and $\mE_n \in \reals^{n \times p}$.
    The number of factors, $k_0$, is fixed.
\end{assumption}

\begin{assumption}
    The matrices of left and right factors, $\mU_n$ and $\mV_n$, have 
    orthonormal columns, i.e. 
    $\mU_n^\trans \mU_n = \mV_n^\trans \mV_n = \mI_{k_0}$.
    Their columns are denoted by
    $\vu_{n,1}, \vu_{n,2}, \ldots, \vu_{n,k_0}$ and
    $\vv_{n,1}, \vu_{n,2}, \ldots, \vv_{n,k_0}$, respectively.
\end{assumption}

\begin{assumption}
    The matrix of factor strengths is diagonal:
    \begin{equation}
        \mD_n = \diag( d_{n,1}, d_{n,2}, \ldots, d_{n,k_0}).  
    \end{equation}
    The strengths
    converge as $d_{n,i}^2 \toas \mu_i$ and 
    $d_{n,i}^2 - \mu_i = \OhP\left( \frac{1}{\sqrt{n}} \right)$, strictly
    ordered as
    \(
        \mu_1 > \mu_2 > \cdots > \mu_{k_0} > 0.
    \)
\end{assumption}

\begin{assumption}\label{A:optimal-leave-out-noise}
    The noise matrix $\mE_n$ is independent of $\mU_n$, $\mD_n$, and $\mV_n$.
    Its elements are iid with 
    \(
        E_{n,11} \sim \Normal( 0, \, \sigma^2 ).
    \)
\end{assumption}

\noindent
These assumptions are standard for latent factor models.

We can apply the work of
Chapter~\ref{C:intrinsic-rank} to get the behavior of the model error.
    We
let $\mX_n = \sqrt{n} \, \mhU_n \mhD_n \mhV_n^\trans$ be the SVD of $\mX_n$ and let $\mhX_n(k) = \sqrt{n} \, \mhU_n \mhD_n(k) \mhV_n^\trans$ be its truncation to $k$ terms.  Then the model error is
\begin{equation}
    {\ME}_n (k)
        = 
            \frac{1}{n \, p} 
            \big\| 
                \sqrt{n} \, \mU_n \mD_n \mV_n^\trans 
                - 
                \mhX_n(k)
            \big\|_\Frob^2.
\end{equation}
With Assumtions~\ref{A:optimal-leave-out-shapes}--\ref{A:optimal-leave-out-noise}, we can apply Proposition~\ref{P:frob-loss-behavior} to get that for fixed
$k$ as $n \to \infty$,
\begin{equation}
    p \cdot {\ME}_n (k)
        \toas
        \sum_{i=1}^{k}
            \alpha_i \mu_i
        +
        \sum_{i=k+1}^{k_0}
            \mu_i
        +
        \sigma^2
        \left(
            1 + \frac{1}{\sqrt{\gamma}}
        \right)^2
        \cdot
        (k - k_0)_+,
\end{equation}
where
\begin{equation}\label{E:bcv-theory-alpha}
    \alpha_i 
    =
    \begin{cases}
        \frac{\sigma^2}{\gamma \mu_i^2}
                \big(
                    3 \sigma^2 + (\gamma+1) \mu_i
                \big)
            &\text{if $\mu_i > \frac{\sigma^2}{\sqrt{\gamma}}$,} \\
        1 
        + 
        \frac{\sigma^2}{\mu_i}
        \left(
            1
            +
            \frac{1}{\sqrt{\gamma}}
        \right)^2
            &\text{otherwise.}
    \end{cases}
\end{equation}
Defining $k_n^\ast$ as the minimizer of ${\ME}_n(k)$, we also get that
\begin{equation}
    k_n^\ast
        \toas
            \max \left\{ i : \mu_i > \mu_\text{crit} \right\},
\end{equation}
where
\begin{equation}
    \mu_\text{crit}
    \equiv
    \sigma^2
    \left(
    \frac{1 + \gamma^{-1}}{2}
    +
    \sqrt{ 
        \left( \frac{1 + \gamma^{-1} }{2} \right)^2
        +
        \frac{3}{\gamma}
    }
    \right),
\end{equation}
provided no $\mu_i$ is exactly eqaul to $\mu_\text{crit}$.  We therefore know
how ${\ME}_n(k)$ and its minimizer behave.

To study the bi-cross-validation estimate of prediction error, we need to 
introduce some more assumptions and notation.  As we are only analyzing first-order 
behavior, we can restrict our analysis to the prediction error estimate from a 
single fold.  We let $\mP_n \in \reals^{n \times n}$ and 
$\mQ_n \in \reals^{p \times p}$ be permutation matrices for the fold, partitioned as
\(
    \mP_n 
        = 
        \left( 
        \begin{matrix}
            \mP_{n,1} & \mP_{n,2}
        \end{matrix}
        \right)
\)
and
\(
    \mQ_n
        =
        \left(
        \begin{matrix}
            \mQ_{n,1} & \mQ_{n,2}
        \end{matrix}
        \right),
\)
with $\mP_{n,1} \in \reals^{n \times n_1}$, $\mP_{n,2} \in \reals^{n \times n_2}$, $\mQ_{n,1} \in \reals^{p \times p_1}$, and $\mQ_{n,2} \in \reals^{p \times p_2}$.  Note that $n = n_1 + n_2$ and that $p = p_1 + p_2$.  We define
$\mX_{n,ij} = \mP_{n,i}^\trans \mX \mQ_{n,j}$,
$\mE_{n,ij} = \mP_{n,i}^\trans \mE \mQ_{n,j}$,
$\mU_{n,i} = \mP_{n,i}^\trans \mU_n$, and
$\mV_{n,j} = \mQ_{n,j}^\trans \mV_n$.  Then in block form,
\begin{align}
    \mP_n^\trans \mX_n \mQ_n
        &=
            \left(
            \begin{matrix}
                \mX_{n,11} & \mX_{n,12} \\
                \mX_{n,21} & \mX_{n,22}
            \end{matrix}
            \right) \notag \\
        &=
            \sqrt{n}
            \left(
            \begin{matrix}
                \mU_{n,1} \mD_n \mV_{n,1}^\trans &
                    \mU_{n,1} \mD_n \mV_{n,2}^\trans \\
                \mU_{n,2} \mD_n \mV_{n,1}^\trans &
                    \mU_{n,2} \mD_n \mV_{n,2}^\trans
            \end{matrix}
            \right)
            +
            \left(
            \begin{matrix}
                \mE_{n,11} & \mE_{n,12} \\
                \mE_{n,21} & \mE_{n,22}
            \end{matrix}
            \right)
\end{align}
This is the starting point of our analysis.

Now we look at the estimate of prediction error.  We let
$\mX_{n,11} = \sqrt{n} \, \mhU_{n,1} \mhD_{n,1} \mhV_{n,1}^\trans$ be the
SVD of $\mX_{n,11}$.  Here,
\begin{subequations}
\begin{align}
    \mhU_{n,1}
        &=
            \left(
            \begin{matrix}
                \vhu_{n,1}^{(1)} & 
                \vhu_{n,2}^{(1)} & 
                \cdots & 
                \vhu_{n,n_1 \wedge p_1}^{(1)}
            \end{matrix}
            \right), \\
    \mhV_{n,1}
        &=
            \left(
            \begin{matrix}
                \vhv_{n,1}^{(1)} & 
                \vhv_{n,2}^{(1)} & 
                \cdots & 
                \vhv_{n,n_1 \wedge p_1}^{(1)}
            \end{matrix}
            \right),
\intertext{and}
    \mhD_{n,1}
        &=
            \diag \left(
                \hd_{n,1}^{(1)},
                \hd_{n,2}^{(1)},
                \ldots,
                \hd_{n,n_1 \wedge p_1}^{(1)}
            \right).
\end{align}
\end{subequations}
For convenience, we define $\hmu_{n,i}^{(1)} = \big(\hd_{n,i}^{(1)}\big)^2$.
For $0 \leq k \leq n_1 \wedge p_1$, we let
\begin{equation}
    \mhD_{n,1}(k)
        =
            \diag \left(
                \hd_{n,1}^{(1)},
                \hd_{n,2}^{(1)},
                \ldots,
                \hd_{n,k}^{(1)},                
                0,
                0,
                \ldots,
                0
            \right)
\end{equation}
so that 
\(
    \mhX_{n,11} (k)
        \equiv \sqrt{n} \, \mhU_{n,1} \mhD_{n,1}(k) \mhV_{n,1}^\trans
\)
is the SVD of $\mX_{n,11}$ truncated to $k$ terms.  This matrix has pseudo-inverse
\(
    \mhX_{n,11}(k)^+
        =
            \frac{1}{\sqrt{n}} \, 
            \mhV_{n,1} \mhD_{n,1}(k)^+ \mhU_{n,1}^\trans.
\)
Therefore, the BCV rank-$k$ prediction of $\mX_{22}$ is
\begin{align}
    \mhX_{n,22}(k)
        &=
            \mX_{n,21} \mhX_{n,11}^{+}(k) \mX_{n,12} \notag \\
        &=
            \big(
                \sqrt{n} \, \mU_{n,2} \mD_n \mV_{n,1}^\trans 
                + \mE_{n,21}
            \big)
            \big(
                \mhX_{n,11}(k)^+
            \big)
            \big(
                \sqrt{n} \, \mU_{n,1} \mD_n \mV_{n,2}^\trans 
                + \mE_{n,12}
            \big) \notag \\
    \begin{split}
        &= 
            \sqrt{n} \, \mU_{n,2} \mD_n \mV_{n,1}^\trans
                \mhV_{n,1} \mhD_{n,1}(k)^+ \mhU_{n,1}^\trans
                \mU_{n,1} \mD_n \mV_{n,2}^\trans \\
        &\quad+
            \mU_{n,2} \mD_n \mV_{n,1}^\trans
                \mhV_{n,1} \mhD_{n,1}(k)^+ \mhU_{n,1}^\trans
                \mE_{n,12} \\
        &\quad+
            \mE_{n,21}
                \mhV_{n,1} \mhD_{n,1}(k)^+ \mhU_{n,1}^\trans
                \mU_{n,1} \mD_n \mV_{n,2}^\trans \\
        &\quad+
            \frac{1}{\sqrt{n}} \, \mE_{n,21}
                \mhV_{n,1} \mhD_{n,1}(k)^+ \mhU_{n,1}^\trans
                \mE_{n,12}
    \end{split} \notag \\
    \begin{split}
        &=
            \sqrt{n} \, 
                \mU_{n,2} \mD_n 
                \mTheta_{n,1} \mhD_{n,1}(k)^+ \mPhi_{n,1}^\trans
                \mD_n \mV_{n,2}^\trans \\
        &\quad+
            \mU_{n,2} \mD_n \mTheta_{n,1} \mhD_{n,1}(k)^+ \mtE_{n,12} \\
        &\quad+
            \mtE_{n,21} 
                \mhD_{n,1}(k)^+ \mPhi_{n,1}^\trans \mD_n \mV_{n,2}^\trans \\
        &\quad+
            \frac{1}{\sqrt{n}} \, \mtE_{n,21} \mhD_{n,1}(k)^+ \mtE_{n,12},
    \end{split}
\end{align}
where 
$\mTheta_{n,1} = \mV_{n,1}^\trans \mhV_{n,1}$,
$\mPhi_{n,1}   = \mU_{n,1}^\trans \mhU_{n,1}$,
$\mtE_{n,12} = \mhU_{n,1}^\trans \mE_{n,12}$, and
$\mtE_{n,21} = \mE_{n,21} \mhV_{n,1}$.  Note that $\mtE_{n,12}$ and $\mtE_{n,21}$ have iid $\Normal( 0 ,\, \sigma^2 )$ entries, also independent of the other terms that make up $\mhX_{n,22}(k)$ and $\mX_{n,22}$.  By conditioning on $\mtE_{n,12}$ and $\mtE_{n,21}$, we can see that $\mhX_{22}(k)$ is in general a biased estimate of $\sqrt{n} \, \mU_{n,2} \mD_n \mV_{n,2}^\trans$.

To analyze $\mhX_{22}(k)$, we need to impose some additional assumptions.
The first assumption is fairly banal and involves the leave-out sizes.

\begin{assumption}
    There exist fixed $K, L \in (0, \infty)$ (not necessarily integers),
    such that
    $\frac{n}{n_2} = K + \oh\left(\frac{1}{\sqrt{n}}\right)$ and 
	$\frac{p}{p_2} = L + \oh\left(\frac{1}{\sqrt{n}}\right)$.
\end{assumption}

\noindent
The next assumption is not quite so innocent and involves the distribution of the factors.

\begin{assumption}\label{A:held-in-orthog}
    The departure from orthogonality for the held-in factors $\mU_{n,1}$
    and $\mV_{n,1}$ is of order $\frac{1}{\sqrt{n}}$.  Specifically,
    \begin{gather*}
        \sup_n 
        \E \, \Big\| 
            \sqrt{n_1} 
            \Big( 
                \mU_{n,1}^\trans \mU_{n,1} 
                -
                \frac{n_1}{n} \mI_{k_0}
            \Big)
        \Big\|_\Frob^2
        <
        \infty, \quad\text{and} \\
        \sup_n
        \E \, \Big\| 
            \sqrt{p_1} 
            \Big( 
                \mV_{n,1}^\trans \mV_{n,1} 
                -
                \frac{p_1}{p} \mI_{k_0}
            \Big)
        \Big\|_\Frob^2
        <
        \infty.
    \end{gather*}
\end{assumption}

\noindent
This assumption is there so that we can apply the theory in Chapter~\ref{C:svd-behavior} to get at the behavior of the SVD of $\mX_{n,11}$.  It is satisfied, for example, if we are performing rotated cross-validation (see subsection~\ref{SS:rotated-cv}) or if the factors are generated by certain stationary processes.  Proposition~\ref{P:project-kframe} in Appendix~\ref{A:projections} is helpful for verifying Assumption~\ref{A:held-in-orthog}.  It is likely that the theory presented below holds under a weaker condition, but a detailed analysis is beyond the scope of this chapter.


\section{Main results}\label{S:bcv-theory-results}

The BCV estimate of prediction error from a single replicate is given
by
\begin{equation}
	\widehat{\PE}_n(k)
		=
			\frac{1}{n_2 \, p_2}
			\| \mX_{n,22} - \mhX_{n,22}(k) \|_\Frob^2.
\end{equation}
It turns out that
\(
	\E\left[
		\widehat{\PE}_n(k)
	\right]
\)
is dominated by the irreducible error.  However, if we have a $\sqrt{n p}$-consistent estimate of $\sigma^2$, then we can get an expression for
the scaled limit of the estimated model error.  This expression is the main result of the chapter.

\begin{theorem}\label{T:bcv-expected-me}
	Suppose that $\hat \sigma_n^2$ is a sequence of
	$\sqrt{n p}$-consistent estimators of $\sigma^2$ satisfying
	\(
		\E\left[
			\sqrt{n p}
				\left(
				\hat \sigma_n^2
				-
				\sigma^2
			\right)
		\right]
		\to
		0.
	\)
	Define the BCV estimate of model error from a single replicate as
	\begin{equation}
		\widehat{\ME}_n(k)
			=
				\widehat{\PE}_n(k)
				-
				\hat \sigma_n^2.
	\end{equation}
	Then, for fixed $k$ as $n\to\infty$,
	\begin{equation}
		\E \left[ p \cdot \widehat{\ME}_n(k) \right]
			\to
				\sum_{i=1}^{k \wedge k_0}
					\beta_i \, \mu_i
				+
				\sum_{i=k+1}^{k_0}
					\mu_i
				+
				\eta
				\cdot
				(k - k_0)_+,
	\end{equation}
	where
	\begin{subequations}
	\begin{align}
		\beta_i
			&=
				\begin{cases}
					\frac{
			            \frac{\sigma^2}{\gamma \mu_{i}^2}
			            \left(
			                3 \sigma^2
			                +
							\left(
								\gamma
								\frac{K-1}{K}
								+
								\frac{L-1}{L}
							\right)
			                \mu_{i}
			            \right)
			        }
			        {
			            \rho
			            +
						\left(
							\gamma
							\frac{K-1}{K}
							+
							\frac{L-1}{L}
						\right)
			            \frac{\sigma^2}{\gamma \mu_{i}}
			            +
			            \frac{\sigma^4}{\gamma \mu_{i}^2}
			        }&\text{when $\mu_{i} > 
								  \frac{\sigma^2}{\sqrt{\rho \, \gamma}}$,} \\
					1 + \frac{\eta}{\mu_i}&\text{otherwise,}
				\end{cases}
	\end{align}
	\begin{gather}
		\rho
			=
				\frac{K-1}{K}
				\cdot
				\frac{L-1}{L}, \\
    \intertext{and}
		\eta
			=
				\frac{\sigma^2}
				     {\left(
					      \sqrt{\gamma}
						  +
						  \sqrt{\frac{K}{K-1}} 
						  \cdot 
						  \sqrt{\frac{L-1}{L}}
					  \right)^{2}
					 }.
	\end{gather}
    \end{subequations}
\end{theorem}

It is interesting to compare $\beta_i$ with the expression for $\alpha_i$ in equation~\eqref{E:bcv-theory-alpha}, which appears in the scaled limit of the true model error, $p \cdot \ME_n(k)$.  Although the BCV 
estimator of model error is biased, this bias is small for large 
$\mu_i$, $K$, and $L$.

A corollary gives the behavior of the minimizer of the expected model 
error estimate. We let $\hat k_n$ be the rank that minimizes 
$\E\left[ \widehat{\ME}_n(k) \right]$.  Note that $\hat k_n$ is a deterministic quantity.  The next two results follow from
Theorem~\ref{T:bcv-expected-me}.

\begin{corollary}\label{C:bcv-expected-me-cutoff}
	As $n \to \infty$,
	\begin{equation}
		\hat k_n
			\to
				\max\left\{
					i :
						\mu_i 
						> 
						\frac{\sigma^2}{\sqrt{\gamma}}
						\cdot
						\sqrt{\frac{2}{\rho}}
				\right\},
	\end{equation}
	(provided no $\mu_i$ is exactly equal to
	\(
		\frac{\sigma^2}{\sqrt{\gamma}}
		\cdot
		\sqrt{\frac{2}{\rho}},
	\)
	in which case the limit is ambiguous).
\end{corollary}

We can use Corollary~\ref{C:bcv-expected-me-cutoff} to guide our choice
of $K$ and $L$.  If we choose them carefully, then $\hat k_n$ and
$k^\ast_n$ will converge to the same value.

\begin{corollary}
	If
	\begin{equation}
		\sqrt{\rho}
			=
				\frac{\sqrt{2}}
					 {\sqrt{\bar \gamma} + \sqrt{\bar \gamma + 3}},
	\end{equation}
	where
	\begin{equation}
		\bar \gamma
			=
				\left(
					\frac{\gamma^{1/2} + \gamma^{-1/2}}{2}
				\right)^2,
	\end{equation}
	then $\hat k_n$ and $k^\ast_n$ converge to the same value
	(provided no $\mu_i$ is exactly equal to
		\(
			\frac{\sigma^2}{\sqrt{\gamma}}
			\cdot
			\sqrt{\frac{2}{\rho}}).
		\)
\end{corollary}

Interestingly, the first-order-optimal choices of $K$ and $L$ do not
depend on the aspect ratio of the original matrix.  All that matters
is the ratio of the number of elements in $\mX_{n,11}$ to the number
of elements in $\mX_{n}$.  

For a square matrix, $\gamma = \bar \gamma = 1$, and the optimal 
$\rho$ is $\frac{2}{9}$.  If we choose $K = L$, then this requires
\[
	\frac{K-1}{K}
		=
			\frac{\sqrt{2}}{3},
\]
so that
\[
	K
		=
			\left(
				1
				-
				\frac{\sqrt{2}}{3}
			\right)^{-1}
	\approx
			1.89.
\]
For general aspect ratios ($\gamma \neq 1$), this requires
\begin{align*}
	K
		&=
			\frac{3}{3
					 -
					 2
					 \left(
						 \sqrt{\bar \gamma + 3}
						 -
					     \sqrt{\bar \gamma}
					 \right)}.
\end{align*}
For very large or very small aspect ratios ($\gamma \to 0$ or $\gamma \to \infty$), $K \to 1$.  In these situations one should leave out almost all of the matrix when performing
bi-cross-validation.

The remainder of the chapter is devoted to proving Theorem~\ref{T:bcv-expected-me}.


\section{The SVD of the held-in block}\label{S:bcv-theory-holdin}

The first step in analyzing the BCV estimate of prediction error is to see how the SVD of $\mX_{n,11}$ behaves.  With Assumptions~\ref{A:optimal-leave-out-shapes}--\ref{A:held-in-orthog}, we can start this analysis.  Our strategy is to use Assumption~\ref{A:held-in-orthog} to apply a matrix perturbation argument in combination with Theorems~\ref{T:spiked-eigenvalue-limits} and~\ref{T:spiked-eigenvector-limits}.

We show that we can apply Theorem~\ref{T:spiked-eigenvector-limits} to 
$\sqrt{n} \, \mU_{n,1} \mD_n \mV_{n,1}^\trans + \mE_{n,11}$ even though $\mU_{n,1}$ and $\mV_{n,1}$ do not have orthogonal columns.  First we set
\begin{equation}
	\gamma_1
		= 
		\frac{K-1}{K}
		\cdot
		\frac{L}{L-1}
		\cdot
		\gamma
\end{equation}
and note that 
\(
	\frac{n_1}{p_1} 
		= 
		\frac{n_1}{n} \cdot \frac{p}{p_1} \cdot \frac{n}{p}
		=
		\gamma_1 + \oh\left( \frac{1}{\sqrt{n_1}} \right).
\)
Next, for $1 \leq i \leq k_0$, we define
\begin{subequations}
\begin{gather}
	\mu_{1,i}
		= 
			\frac{L-1}{L} \cdot \mu_{i}, \\
	\bmu_{1,i}
		=
		\begin{cases}
			\frac{K-1}{K}
			\cdot
            \left( \mu_{1,i} + \sigma^2 \right)
            \left( 1 + \frac{\sigma^2}{\gamma_1 \mu_{1,i}} \right)
                &\text{when $\mu_{1,i} 
							 > 
							 \frac{\sigma^2}{\sqrt{\gamma_1}}$}, \\
			\frac{K-1}{K}
			\cdot
            \sigma^2 \left( 1 + \frac{1}{\sqrt{\gamma_1}} \right)^2
                &\text{otherwise.}
        \end{cases}		
			\\
	\theta_{1,i}
		=
	        \begin{cases}
				\sqrt{\frac{L-1}{L}}
				\cdot
				\sqrt{
		            \left( 
						1 
						- 
						\frac{\sigma^4}{ \gamma_1 \mu_{1,i}^2}
					\right)
		            \left(
						1 
						+ 
						\frac{\sigma^2}{ \gamma_1 \mu_{1,i} } 
					\right)^{-1}
				}
	            &\text{when $\mu_{1,i} 
							 > 
							 \frac{\sigma^2}{\sqrt{\gamma_1}}$,} \\
	            0
	            &\text{otherwise,}
	        \end{cases} \\
    \varphi_{1,i}
        =
            \begin{cases}
				\sqrt{\frac{K-1}{K}}
				\cdot
				\sqrt{
	                \left( 
						1 
						- 
						\frac{\sigma^4}{ \gamma_1 \mu_{1,i}^2} 
					\right)
	                \left(
						1 
						+ 
						\frac{\sigma^2}{ \mu_{1,i} } 
					\right)^{-1}
				}
                &\text{when $\mu_{1,i} 
							 > 
							\frac{\sigma^2}{\sqrt{\gamma_1}}$,} \\
                0
                &\text{otherwise.}
            \end{cases}
\end{gather}
\end{subequations}
For $i > k_0$, we put 
\(
    \bmu_{1,i} 
        =
            \frac{K-1}{K} 
            \cdot
            \sigma^2
            \left(
                1
                +
                \frac{1}{\sqrt{\gamma_1}}
            \right)^2.
\)
Now, for $k \geq 1$ we let $\mTheta_{1}(k)$ and $\mPhi_1(k)$ be $k_0 \times k$ matrices with entries
\begin{equation}
    \theta_{1,ij}^{(k)}
        =
            \begin{cases}
                \theta_{1,i}&\text{if $i = j$,} \\
                0&\text{otherwise,}
            \end{cases}
    \qquad
    \text{and}
    \qquad
    \varphi_{1,ij}^{(k)}
        =
            \begin{cases}
                \varphi_{1,i}&\text{if $i = j$,} \\
                0&\text{otherwise,}
            \end{cases}
\end{equation}
respectively.  In this section, we prove the following:

\begin{proposition}\label{P:holdin-svd}
    For fixed $k$ as $n\to\infty$, the first $k$ columns of $\mTheta_{1,n}$
    and $\mPhi_{1,n}$ converge in probability to $\mTheta_{1}(k)$ and 
    $\mPhi_{1}(k)$, respectively.  Likewise, for $1 \leq i \leq k$,
    $\hd_{n,i}^{(1)} \toP \bmu_{1,i}^{1/2}$.
\end{proposition}

We prove the proposition by leveraging the work of 
Chapter~\ref{C:svd-behavior}.  We have that 
\(
    \mX_{n,11} = \sqrt{n} \, \mU_{n,1} \mD_n \mV_{n,1}^\trans + \mE_{n,11}.
\)
The first term does not satisfy the conditions of Theorems~\ref{T:spiked-eigenvalue-limits}~and~\ref{T:spiked-eigenvector-limits} since $\mU_{n,1}$ and $\mV_{n,1}$ do not have orthonormal columns.  Moreover, the scaling is $\sqrt{n}$ instead of $\sqrt{n_1}$.  We introduce scaling constants and group the terms as
\begin{equation}
    \mX_{n,11}
        =
            \sqrt{n_1}
            \Big(
                \sqrt{\frac{n}{n_1}} \mU_{n,1}
            \Big)
            \Big(
                \sqrt{\frac{p_1}{p}} \mD_n
            \Big)
            \Big(
                \sqrt{\frac{p}{p_1}}
                \mV_{n,1}
            \Big)^\trans
            +
            \mE_{n,11}.
\end{equation}
With this scaling,
\[
    \mE \bigg[
        \Big( \sqrt{\frac{n}{n_1}} \mU_{n,1} \Big)^\trans
        \Big( \sqrt{\frac{n}{n_1}} \mU_{n,1} \Big)
    \bigg]
    =
    \mE \bigg[
        \Big( \sqrt{\frac{p}{p_1}} \mV_{n,1} \Big)^\trans
        \Big( \sqrt{\frac{p}{p_1}} \mV_{n,1} \Big)
    \bigg]
    =
    \mI_{k_0},
\]
and the diagonal elements of
\(
    \Big(
          \sqrt{\frac{p_1}{p}} \mD_n
    \Big)
\)
converge to 
\(
    \mu_{1,1}^{1/2},
    \mu_{1,2}^{1/2},
    \ldots,
    \mu_{1,k_0}^{1/2}.
\)
So, Proposition~\ref{P:holdin-svd} should at least be plausible.  

We prove the result by showing that
\(
    \Big(
        \sqrt{\frac{n}{n_1}} \mU_{n,1}
    \Big)
    \Big(
        \sqrt{\frac{p_1}{p}} \mD_n
    \Big)
    \Big(
        \sqrt{\frac{p}{p_1}}
        \mV_{n,1}
    \Big)^\trans
\)
is \emph{almost} an SVD.  We denote its $k_0$-term SVD by 
$\mtU_{n,1} \mtD_{n,1} \mtV_{n,1}^\trans$, and demonstrate the following:
\begin{lemma}\label{L:svd-random-perturb}
    Three properties hold:
    \begin{enumerate}[(1)]
        \item For $1 \leq i \leq k_0$, 
            \(
                |\frac{p_1}{p} d_{n,i}^2 - \tilde d_{n,i}^2| 
                    = 
                        \OhP\left(\frac{1}{\sqrt{n}}\right).
            \)
        \item For any sequence of vectors $\vx_1, \vx_2, \ldots, \vx_n$
            with $\vx_n \in \reals^n$,
            \[
                \left\|
                    \Big(
                        \sqrt{\frac{n}{n_1}} \mU_{n,1}
                    \Big)^\trans
                    \vx_n
                    -
                    \mtU_{n,1}^\trans \,
                    \vx_n
                \right\|_2
                =
                \OhP\left(\frac{\| \vx_n \|_2}{\sqrt{n}}\right).
            \]
        \item For any sequence of vectors $\vy_1, \vy_2, \ldots, \vy_n$
            with $\vy_n \in \reals^p$,
            \[
                \left\|
                    \Big(
                        \sqrt{\frac{p}{p_1}} \mV_{n,1}
                    \Big)^\trans
                    \vy_n
                    -
                    \mtV_{n,1}^\trans \,
                    \vy_n
                \right\|_2
                =
                \OhP\left(\frac{\| \vy_n \|_2}{\sqrt{n}}\right).
            \]
    \end{enumerate}
    Above, $\tilde d_{n,i}$ is the $i$th diagonal entry of $\mtD_{n}$,
    which are assumed to be sorted in descending order.
\end{lemma}
\noindent
Proposition~\ref{P:holdin-svd} is then a direct consequence of
Lemma~\ref{L:svd-random-perturb} combined with
Theorems~\ref{T:spiked-eigenvalue-limits}~and~\ref{T:spiked-eigenvector-limits}.

\begin{proof}[Proof of Lemma~\ref{L:svd-random-perturb}]
    First, let $\sqrt{\frac{n}{n_1}} \mU_{n,1} = \mbU_{n,1} \mR_n$ be a 
    $QR$-decomposition so that $\mbU_{n,1} \in \reals^{n_1 \times k_0}$ has
    orthonormal columns and $\mR_n$ is an upper-triangular matrix.  Define
    $\mR_{n,1} = \sqrt{n} ( \mR_{n,1} - \mI_{k_0})$ so that
    $\mR_n = \mI_{k_0} + \frac{1}{\sqrt{n}} \mR_{n,1}$.  We can write
    \[
        \frac{n}{n_1}
        \mU_{n,1}^\trans \mU_{n,1}
            =
                \mI_{k_0}
                +
                \frac{1}{\sqrt{n}}
                \left(
                    \mR_{n,1}
                    +
                    \mR_{n,1}^\trans
                \right)
                +
                \frac{1}{n}
                \mR_{n,1}^\trans \mR_{n,1}.
    \]
    By Assumption~\ref{A:held-in-orthog},
    \(
        \frac{n}{n_1} \mU_{n,1}^\trans \mU_{n,1}
        -
        I_{k_0}
        =
        \OhP\left( \frac{1}{\sqrt{n}} \right).
    \)
    Therefore, $\mR_{n,1} = \OhP\left( 1 \right)$.
    
    The same argument applies to show that there exits a
    $\mbV_{n,1} \in \reals^{p_1 \times k_0}$ with orthonormal columns
    such that
    \[
        \sqrt{\frac{p}{p_1}} \mV_{n,1} 
            =
            \mtV_{n,1}
            \left(
                \mI_{k_0}
                +
                \frac{1}{\sqrt{n}}
                \mS_{n,1}
            \right),
    \]
    and $\mS_{n,1}$ is upper-triangular with $\mS_{n,1} = \OhP(1)$.
    
    We now look at
    \begin{multline*}
        \Big(
            \mI_{k_0}
            +
            \frac{1}{\sqrt{n}}
            \mR_{n,1}
        \Big)
        \Big(
            \mD_n
        \Big)
        \Big(
            \mI_{k_0}
            +
            \frac{1}{\sqrt{n}}
            \mS_{n,1}
        \Big)^\trans \\
            =
                \mD_n
                +
                \frac{1}{\sqrt{n}}
                \left(
                    \mR_{n,1} \mD_n
                    +
                    \mD_n \mS_{n,1}^\trans
                \right)
                +
                \frac{1}{n}
                \mR_{n,1} \mD_n \mS_{n,1}^\trans.
    \end{multline*}
    Since the diagonal elements of $\mD_n$ are distinct, we can apply
    Lemma~\ref{L:eigen-perturb} twice to get that there exist
    $k_0 \times k_0$ matrices $\mbR_{n,1}$, $\mbS_{n,1}$ and $\mDelta_n$
    of size $\OhP(1)$ such that
    \begin{multline*}
        \Big(
            \mI_{k_0}
            +
            \frac{1}{\sqrt{n}}
            \mR_{n,1}
        \Big)
        \Big(
            \mD_n
        \Big)
        \Big(
            \mI_{k_0}
            +
            \frac{1}{\sqrt{n}}
            \mS_{n,1}
        \Big)^\trans \\
            =
                \Big(
                    \mI_{k_0}
                    +
                    \frac{1}{\sqrt{n}}
                    \mbR_{n,1}
                \Big)
                \Big(
                    \mD_n
                    +
                    \frac{1}{\sqrt{n}}
                    \mDelta_n
                \Big)
                \Big(
                    \mI_{k_0}
                    +
                    \frac{1}{\sqrt{n}}
                    \mbS_{n,1}
                \Big)^\trans,
    \end{multline*}
    with 
    \(
        \mI_{k_0}
        +
        \frac{1}{\sqrt{n}}
        \mbR_{n,1}
    \)
    and
    \(
        \mI_{k_0}
        +
        \frac{1}{\sqrt{n}}
        \mbS_{n,1}
    \)
    both orthogonal matrices, and $\mDelta_n$ diagonal.
    
    We define
    \(
        \mtU_{n,1} 
            = 
            \mbU_{n,1}
            \Big(
                \mI_{k_0}
                +
                \frac{1}{\sqrt{n}}
                \mbR_{n,1}
            \Big),
    \)
    \(
        \mtV_{n,1} 
            = 
            \mbV_{n,1}
            \Big(
                \mI_{k_0}
                +
                \frac{1}{\sqrt{n}}
                \mbS_{n,1}
            \Big),
    \)
    and
    \(
        \mtD_n
            =
            \sqrt{\frac{p_1}{p}}
            \Big(
                \mD_n
                +
                \frac{1}{\sqrt{n}}
                \mDelta
            \Big).
    \)
    Now, as promised, $\mtU_{n,1} \mtD_n \mtV_{n,1}^\trans$ is the
    SVD of
    \[
        \Big(
            \sqrt{\frac{n}{n_1}} \mU_{n,1}
        \Big)
        \Big(
            \sqrt{\frac{p_1}{p}} \mD_n
        \Big)
        \Big(
            \sqrt{\frac{p}{p_1}}
            \mV_{n,1}
        \Big)^\trans.
    \]
    We can see immediately the property (1) holds.  For property (2), note
    that
    \begin{align*}
        \mtU_{n,1}
            &=
                \mbU_{n,1}
                \Big(
                    \mI_{k_0}
                    +
                    \frac{1}{\sqrt{n}}
                    \mbR_{n,1}
                \Big) \\
            &= 
                \sqrt{\frac{n}{n_1}}                
                \mU_{n,1}
                \Big(
                    \mI_{k_0}
                    +
                    \frac{1}{\sqrt{n}}
                    \mR_{n,1}
                \Big)^{-1}
                \Big(
                    \mI_{k_0}
                    +
                    \frac{1}{\sqrt{n}}
                    \mbR_{n,1}
                \Big) \\
            &= 
                \sqrt{\frac{n}{n_1}}
                \mU_{n,1}
                \Big(
                    \mI_{k_0}
                    +
                    \frac{1}{\sqrt{n}}
                    \mtR_{n,1}
                \Big)
    \end{align*}
    for some $\mtR_{n,1} = \OhP(1)$.  Therefore,
    \[
        \mtU_{n,1}^\trans \vx_n - \mU_{n,1}^\trans \vx_n
            =
                \frac{1}{\sqrt{n}}
                \mtR_{n,1}^\trans \mU_{n,1}^\trans \vx_n
    \]
    so that
    \begin{align*}
        \| \mtU_{n,1}^\trans \vx_n - \mU_{n,1}^\trans \vx_n \|_2
            &\leq 
                \frac{1}{\sqrt{n}}
                \| \mtR_{n,1} \|_\Frob
                \cdot
                \| \mU_{n,1} \|_\Frob
                \cdot
                \| \vx_n \|_2 \\
            &=
                \OhP \left( \frac{\| \vx_n \|_2 }{\sqrt{n}} \right).
    \end{align*}
    A similar argument applies to show that property (3) holds.
\end{proof}

\section{The prediction error estimate}\label{S:bcv-pe-estimate}

In this section, we study the estimate of prediction error
\[
	\widehat{\PE}_n(k)
		=
			\frac{1}{n_2 \, p_2}
			\| \mX_{n,22} - \mhX_{n,22}(k) \|_\Frob^2
\]
We can expand this as
\begin{align}
	\widehat{\PE}_n(k)
		&=
			\frac{1}{n_2 \, p_2}
			\tr \Big(
				\big(
					\mX_{n,22} - \mhX_{n,22}(k)
				\big)
				\big(
					\mX_{n,22} - \mhX_{n,22}(k)
				\big)^\trans
			\Big) \notag \\
	\begin{split}
		&=
			\frac{1}{n_2 \, p_2}
			\tr \Big(
				\big(
					\sqrt{n} \, \mU_{n,2} \mD_n \mV_{n,2}^\trans 
					+
					\mE_{n,22}
					- 
					\mhX_{n,22}(k)
				\big) \\
				&\qquad\qquad\qquad\cdot
				\big(
					\sqrt{n} \, \mU_{n,2} \mD_n \mV_{n,2}^\trans 
					+
					\mE_{n,22}
					- 
					\mhX_{n,22}(k)
				\big)^\trans
			\Big)
	\end{split} \notag \\
	\begin{split}
		&=
			\frac{1}{n_2 \, p_2}
			\bigg(
				\big\| 
					\sqrt{n} \, \mU_{n,2} \mD_n \mV_{n,2}^\trans
					-
					\mhX_{n,22}(k)
				\big\|_\Frob^2 \\
				&\qquad\qquad\quad+
				2 \, 
				\tr \Big(
					\mE_{n,22}
					\big(
						\sqrt{n} \, \mU_{n,2} \mD_n \mV_{n,2}^\trans 
						- 
						\mhX_{n,22}(k)
					\big)^\trans
				\Big) \\
				&\qquad\qquad\quad+
				\|
					\mE_{n,22}
				\|_\Frob^2
			\bigg).
	\end{split}
\end{align}
It has expectation
\begin{equation}
	\E \left[ \widehat{\PE}_n(k) \right]
		=
			\E \left[
				\frac{1}{n_2 \, p_2}
				\big\| 
					\sqrt{n} \, \mU_{n,2} \mD_n \mV_{n,2}^\trans
					-
					\mhX_{n,22}(k)
				\big\|
			\right]
			+
			\sigma^2.
\end{equation}
The first term is the expected model approximation error and the second term is the irreducible error.

The expected model approximation error expands into four terms.  We have
\begin{multline}
	\E \big\|
		\sqrt{n} \, \mU_{n,2} \mD_n \mV_{n,2}^\trans
		-
		\mhX_{n,22}(k)
	\big\|_\Frob^2 \\
		=
			\E\Big[ 
				\tr \Big(
					\big(
						\sqrt{n} \,
						\mU_{n,2} 
						( 
							\mD_n 
							- 
							\mD_n 
							\mTheta_{n,1} \mhD_{n,1}(k)^+ \mPhi_{n,1}
							\mD_n
						)
						\mV_{n,2}^\trans \\
						-
						\mU_{n,2} 
						\mD_n \mTheta_{n,1} \mhD_{n,1}(k)^+ 
						\mtE_{n,12} \\
						- 
						\mtE_{n,21}
						\mhD_{n,1}(k)^+ \mPhi_{n,1}^\trans \mD_n 
						\mV_{n,2}^\trans \\
						-
						\frac{1}{\sqrt{n}}
						\mtE_{n,21}
						\mhD_{n,1}(k)^+
						\mtE_{n,12}
					\big) \\
					\cdot
					\big(
						\sqrt{n} \,
						\mU_{n,2} 
						( 
							\mD_n 
							- 
							\mD_n 
							\mTheta_{n,1} \mhD_{n,1}(k)^+ \mPhi_{n,1}
							\mD_n
						)
						\mV_{n,2}^\trans \\
						-
						\mU_{n,2} 
						\mD_n \mTheta_{n,1} \mhD_{n,1}(k)^+ 
						\mtE_{n,12} \\
						- 
						\mtE_{n,21}
						\mhD_{n,1}(k)^+ \mPhi_{n,1}^\trans \mD_n 
						\mV_{n,2}^\trans \\
						-
						\frac{1}{\sqrt{n}}
						\mtE_{n,21}
						\mhD_{n,1}(k)^+
						\mtE_{n,12}
					\big)^\trans
				\Big)
			\Big].
\end{multline}
By first conditioning on everything but $\mtE_{n,12}$ and $\mtE_{n,21}$, the
cross-terms cancel and we get
\begin{multline}
	\E \big\|
		\sqrt{n} \, \mU_{n,2} \mD_n \mV_{n,2}^\trans
		-
		\mhX_{n,22}(k)
	\big\|_\Frob^2 \\
		=
			\E\big\|
				\sqrt{n} \,
				\mU_{n,2} 
				( 
					\mD_n 
					- 
					\mD_n 
					\mTheta_{n,1} \mhD_{n,1}(k)^+ \mPhi_{n,1}
					\mD_n
				)
				\mV_{n,2}^\trans 
			\big\|_\Frob^2 \\
			+
			\E\big\|
				\mU_{n,2} 
				\mD_n \mTheta_{n,1} \mhD_{n,1}(k)^+ 
				\mtE_{n,12}
			\big\|_\Frob^2 \\
			+
			\E\big\|
				\mtE_{n,21}
				\mhD_{n,1}(k)^+ \mPhi_{n,1}^\trans \mD_n 
				\mV_{n,2}^\trans
			\big\|_\Frob^2 \\
			+
			\E\big\|
				\frac{1}{\sqrt{n}}
				\mtE_{n,21}
				\mhD_{n,1}(k)^+
				\mtE_{n,12}
			\big\|_\Frob^2.
\end{multline}
Since $\mtE_{n,12}$ and $\mtE_{n,21}$ are made of iid $\Normal( 0, \, \sigma^2)$ random variables, the last three terms are fairly easy to analyze.
We can use the following lemma:

\begin{lemma}\label{L:frob-random-product}
	Let $\mZ \in \reals^{m \times n}$ be a random matrix with uncorrelated
	elements, all having mean $0$ and variance $1$ (but not necessarily 
	from the same distribution).  If $\mA \in \reals^{n \times p}$ is 
	independent of $\mZ$, then
	\[
		\E \left\|
			\mZ \mA
		\right\|_\Frob^2
			=
				m
				\cdot
				\mE \left\| \mA \right\|_\Frob^2.
	\]
\end{lemma}
\begin{proof}
	The square of the $ij$ element of the product is given by
	\begin{align*}
		\left( \mZ \mA \right)_{ij}^2
			&=
				\Big(
					\sum_{\alpha=1}^n
						Z_{i \alpha} A_{\alpha j}
				\Big)^2 \\
			&=
				\sum_{\alpha=1}^n
					Z_{i \alpha}^2 A_{\alpha j}^2
				+
				\sum_{\alpha \neq \beta}
					Z_{i \alpha} A_{\alpha j}
					Z_{i \beta}  A_{\beta j}
	\end{align*}
	This has expectation
	\[
		\E \left[
			\left( \mZ \mA \right)_{ij}^2
		\right]
			=
				\sum_{\alpha=1}^n
					\E\left[ A_{\alpha j}^2 \right],
	\]
	so that
	\begin{align*}
		\E \left\|
			\mZ \mA
		\right\|_\Frob^2
			&=
				\sum_{i=1}^m
				\sum_{j=1}^p
					\E \left[
						\left( \mZ \mA \right)_{ij}^2
					\right] \\
			&=
				\sum_{i=1}^m
				\sum_{j=1}^p
					\sum_{\alpha=1}^n
						\E\left[ A_{\alpha j}^2 \right] \\
			&= m \cdot \E \left\| \mA \right\|_\Frob^2. \qedhere
	\end{align*}
\end{proof}

We also need a technical result to ensure that after appropriate scaling, the
expectations are finite.

\begin{lemma}\label{L:d1-bounded-away-from-0}
	For fixed $k$ as $n \to \infty$, 
	$\hd_{n,k}^{(1)} $ is almost surely bounded away from zero.
\end{lemma}
\begin{proof}
	We have that $\hd_{n,k}^{(1)}$ is the $k$th singular value of
	\(
		\frac{1}{\sqrt{n}} \mX_{n,11}
			=
				\mU_{n,1} \mD_n \mV_{n,1}^\trans
				+
				\frac{1}{\sqrt{n}}
				\mE_{n,11}.
	\)
	This is the same as the $k$th eigenvalue of
	\begin{multline*}
		\frac{1}{n} \mX_{n,11} \mX_{n,11}^\trans
			=
				\mU_{n,1} \mD_n \mV_{n,1}^\trans 
					\mV_{n,1} \mD_n \mU_{n,1}^\trans
				+
				\frac{1}{\sqrt{n}}
				\mU_{n,1} \mD_n \mV_{n,1}^\trans \mE_{n,11}^\trans \\
				+
				\frac{1}{\sqrt{n}}
				\mE_{n,11}
				\mV_{n,1} \mD_n \mU_{n,1}^\trans
				+
				\frac{1}{n}
				\mE_{n,11} \mE_{n,11}^\trans.
	\end{multline*}
	For each $n$, we choose 
	\(
		\mO_n
			=
			\left(
			\begin{matrix}
				\mO_{n,1} & \mO_{n,2}
			\end{matrix}
			\right)
		\in \reals^{n \times n}
	\)
	to be an orthogonal matrix with $\mO_{n,2} \in \reals^{n \times n-k_0}$ 
	and $\mO_{n,2}^\trans \mU_{n,1} = 0$.
	Then, with $\lambda_k(\cdot)$ denoting the $k$th eigenvalue, we have
	\begin{align*}
		\hd_{n,k}^{(1)}
			&=
				\lambda_k \left(
					\frac{1}{n} \mX_{n,11} \mX_{n,11}^\trans
				\right) \\
			&=
				\lambda_k \left(
					\mO_{n}^\trans
					\Big(
						\frac{1}{n} \mX_{n,11} \mX_{n,11}^\trans
					\Big)
					\mO_n
				\right) \\
			&=
				\lambda_k \left(
					\left(
					\begin{matrix}
						\mA_{n,11} & \mA_{n,12} \\
						\mA_{n,21} & \mA_{n,22}
					\end{matrix}
					\right)
				\right),
	\end{align*}
	for
	\(
		\mA_{n,ij} 
			= 
				\frac{1}{n}
				\mO_{n,i}^\trans 
				\mX_{n,11}^\trans \mX_{n,11}
				\mO_{n,j}.
	\)
	Note that
	\(
		\mA_{n,22} 
			= 
				\frac{1}{n} 
				\mO_{n,2}^\trans \mE_{n,11} \mE_{n,11}^\trans \mO_{n,2}
	\)
	is an $(n-k) \times (n-k)$ matrix with iid $\Normal( 0, \, \sigma^2)$
	entries.
	
	Define $G_n = \lambda_k ( \mA_{n,22} )$
	By the eigenvalue interlacing inequality \cite{golub1996mc}, we have
	that
	\(
		\hd_{n,k}^{(1)}
			\geq
				G_n.
	\)
	Moreover, Theorems~\ref{T:mp-limit}~and~\ref{T:max-wishart-eig-limit}
	give us that 
	\(
		G_n
			\toas 
				\sigma^2
				\left(
					1 + \frac{1}{\sqrt{\gamma_1}}
				\right)^2.
	\)
	Hence, almost surely for $n$ large enough 
	$\hd_{n,k}^{(1)} \geq G_n \geq \sigma^2$.
\end{proof}

With these lemmas, we can prove the following:

\begin{lemma}
	The terms in the expected model approximation error converge as:
	\begin{subequations}
	\begin{gather}
		\begin{split}
		\E\bigg[ 
			\frac{p}{n_2 \, p_2} 
			\Big\|
				\sqrt{n} \,
				\mU_{n,2} 
				( 
					\mD_n 
					&- 
					\mD_n 
					\mTheta_{n,1} \mhD_{n,1}(k)^+ \mPhi_{n,1}
					\mD_n
				)
				\mV_{n,2}^\trans 
			\Big\|_\Frob^2
		\bigg] \\
			&\to 
			 	\sum_{i=1}^{k \wedge k_0}
					\mu_i 
					\left( 
						1 
						- 
						\sqrt{ \frac{ \mu_i }{ \bmu_{1,i} } }
						\theta_{1,i}
						\varphi_{1,i}
					\right)^2
				+
				\sum_{i=k+1}^{k_0}
					\mu_i,
		\end{split}\\
		\E\left[ 
			\frac{p}{n_2 \, p_2} 
			\Big\|
				\mU_{n,2} 
				\mD_n \mTheta_{n,1} \mhD_{n,1}(k)^+ 
				\mtE_{n,12}
			\Big\|_\Frob^2
		\right]
			\to
				\frac{\sigma^2}{\gamma}
				\cdot 
				\sum_{i=1}^{k \wedge k_0}
					\frac{\mu_i}{\bmu_{1,i}}
					\theta_{1,i}^2,
				\\
		\E\left[ 
			\frac{p}{n_2 \, p_2} 
			\Big\|
				\mtE_{n,21}
				\mhD_{n,1}(k)^+ \mPhi_{n,1}^\trans \mD_n 
				\mV_{n,2}^\trans
			\Big\|_\Frob^2
		\right]
			\to
				\sigma^2
				\cdot
				\sum_{i=1}^{k \wedge k_0} 
					\frac{\mu_i}{\bmu_{1,i}}
					\varphi_{1,i}^2,
				\\
\intertext{and}
		\E\left[ 
			\frac{p}{n_2 \, p_2} 
			\Big\|
				\frac{1}{\sqrt{n}}
				\mtE_{n,21}
				\mhD_{n,1}(k)^+
				\mtE_{n,12}
			\Big\|_\Frob^2
		\right]
			\to
				\frac{\sigma^2}{\gamma}
				\cdot
				\sum_{i=1}^k
					\frac{\sigma^2}{\bmu_{1,i}}.
	\end{gather}
    \end{subequations}
\end{lemma}
\begin{proof}
	The squared Frobenius norm in the first term is equal to
	\begin{align*}
		\tr\Big(
			&\big(
				\sqrt{n} \,
				\mU_{n,2} 
				( 
					\mD_n 
					- 
					\mD_n 
					\mTheta_{n,1} \mhD_{n,1}(k)^+ \mPhi_{n,1}
					\mD_n
				)
				\mV_{n,2}^\trans 
			\big) \\
			&\qquad\cdot
			\big(
				\sqrt{n} \,
				\mU_{n,2} 
				( 
					\mD_n 
					- 
					\mD_n 
					\mTheta_{n,1} \mhD_{n,1}(k)^+ \mPhi_{n,1}
					\mD_n
				)
				\mV_{n,2}^\trans 
			\big)^\trans
		\Big) \\
		&=
			n
			\cdot
			\tr\Big(
				\big( 
					\mD_n 
					- 
					\mD_n 
					\mTheta_{n,1} \mhD_{n,1}(k)^+ \mPhi_{n,1}
					\mD_n
				\big)
				\cdot
				\big(
					\mV_{n,2}^\trans 
					\mV_{n,2}
				\big) \\
				&\qquad\qquad\cdot
				\big( 
					\mD_n 
					- 
					\mD_n 
					\mTheta_{n,1} \mhD_{n,1}(k)^+ \mPhi_{n,1}
					\mD_n
				\big)
				\cdot
				\big(
					\mU_{n,2}^\trans
					\mU_{n,2} 
				\big) 
			\Big).
	\end{align*}
	Now, 
		$\mU_{n,2}^\trans \mU_{n,2} \toP \frac{1}{K} \mI_{k_0}$,
		$\mV_{n,2}^\trans \mV_{n,2} \toP \frac{1}{L} \mI_{k_0}$,
	and
	\begin{align*}
		\mD_n &- \mD_n \mTheta_{n,1} \mhD_{n,1}(k)^+ \mPhi_{n,1} \mD_n \\
			&\toP
				\diag\Big(
					\mu_1^{1/2}
					-
					\mu_1^{1/2}
					\theta_{1,1}
					\bmu_{1,1}^{-1/2}(k) \,
					\varphi_{1,1}
					\mu_1^{1/2},
					\quad
					\mu_2^{1/2}
					-
					\mu_2^{1/2}
					\theta_{1,2}
					\bmu_{1,2}^{-1/2}(k) \,
					\varphi_{1,2}
					\mu_2^{1/2},
					\quad
					\ldots, \\
					&\qquad\qquad\qquad
					\mu_{k_0}^{1/2}
					-
					\mu_{k_0}^{1/2}
					\theta_{1,k_0}
					\bmu_{1,k_0}^{-1/2}(k) \,
					\varphi_{1,k_0}
					\mu_{k_0}^{1/2} 
				\Big),
	\end{align*}
	where
	\[
		\bmu_{1,i}(k) 
			=
			\begin{cases}
				\bmu_{1,i} &\text{if $i \leq k$,} \\
				0          &\text{otherwise.}
			\end{cases}
	\]
	We can apply the Bounded Convergence Theorem to get the result for the 
	first
	term since the elements of $\mU_{n,2}$, $\mV_{n,2}$, $\mTheta_{n,1}$ 
	and $\mPhi_{n,1}$ are bounded by $1$ and since
	Lemma~\ref{L:d1-bounded-away-from-0} ensures that the elements of
	$\mhD_{n,1}(k)^+$ are bounded as well.
	
	The last three terms can be gotten similarly by applying
	Lemmas~\ref{L:frob-random-product}~and~\ref{L:d1-bounded-away-from-0}.
\end{proof}

We can now get an expression for the limit of the estimated model
approximation error.  Specifically,
\begin{align}
	\begin{split}
	\E \bigg[  
		\frac{p}{n_2 \, p_2}
		\Big\|
			\sqrt{n} \, &\mU_{n,2} \mD_n \mV_{n,2}^\trans
			-
			\mhX_{n,22}(k)
		\Big\|_\Frob^2
	\bigg] \\
		&=
			\sum_{i=1}^{k \wedge k_0}
			\Bigg\{
				\mu_i 
				\left( 
					1
					-
					\sqrt{\frac{\mu_i}{\bmu_{1,i}}}
					\theta_{1,i}
					\varphi_{1,i}
				\right)^2
				+
				\sigma^2
				\frac{\mu_i}{\bmu_i}
				\left(
					\gamma^{-1} \theta_{1,i}^2
					+
					\varphi_{1,i}^2
				\right)
				+
				\frac{\sigma^4}{\gamma \bmu_{1,i}}
			\Bigg\} \\
			&\qquad\qquad+
			\sum_{i=k+1}^{k_0}
				\mu_i
			+
			\frac{\sigma^2}{\gamma}
			\cdot\!\!\!
			\sum_{i=k_0+1}^k
				\frac{\sigma^2}{\bmu_{1,i}}
	\end{split} \notag \\
		&=
			\sum_{i=1}^{k \wedge k_0}
				\beta_i \, \mu_i
			+
			\sum_{i=k+1}^{k_0}
				\mu_i
			+
			\frac{\sigma^2}{\gamma}
			\left(
				1
				+
				\frac{1}{\sqrt{\gamma_1}}
			\right)^{-2}\!\!\!\!\!
			\cdot
			(k - k_0)_+,
\end{align}
where
\begin{align}
	\beta_i
		&=
			\left( 
				1
				-
				\sqrt{\frac{\mu_i}{\bmu_{1,i}}}
				\theta_{1,i}
				\varphi_{1,i}
			\right)^2
			+
			\frac{\sigma^2}{\bmu_{1,i}}
			\left(
				\gamma^{-1} \theta_{1,i}^2
				+
				\varphi_{1,i}^2
			\right)
			+
			\frac{1}{\mu_i}
			\frac{\sigma^4}{\gamma \bmu_{1,i}} \notag \\
		&=
			1
			-
			2
			\sqrt{\frac{\mu_i}{\bmu_{1,i}}}
			\theta_{1,i}
			\varphi_{1,i}
			+
			\frac{\mu_i}{\bmu_{1,i}}
			\theta_{1,i}^2
			\varphi_{1,i}^2
			+
			\frac{\sigma^2}{\bmu_{1,i}}
			\left(
				\gamma^{-1} \theta_{1,i}^2
				+
				\varphi_{1,i}^2
			\right)
			+
			\frac{1}{\mu_i}
			\frac{\sigma^4}{\gamma \bmu_{1,i}}.
\end{align}
If $\mu_{1,i} \leq \frac{\sigma^2}{\sqrt{\gamma_1}}$, then
$\theta_{1,i} = \varphi_{1,i} = 0$ and
\(
	\bmu_{1,i} 
		= 
			\frac{K-1}{K} 
			\cdot 
			\sigma^2
			\left(
				1
				+
				\frac{1}{\sqrt{\gamma_1}}
			\right)^2,
\)
so that
\[
	\beta_i
		=
			1
			+
			\frac{1}{\gamma}
			\cdot
			\frac{\sigma^2}{\mu_i}
			\cdot
			\left(
				1
				+
				\frac{1}{\sqrt{\gamma_1}}
			\right)^{-2}.
\]
In the opposite situation
($\mu_{1,i} > \frac{\sigma^2}{\sqrt{\gamma_1}}$), we define
\[
    \rho
        =
            \frac{L-1}{L}
            \cdot
            \frac{K-1}{K}
\]
and get the simplifications
\begin{gather*}
    \frac{\bmu_{1,i}}{\mu_i}
        =
            \rho
            \cdot
            \left(
                1
                +
                (1 + \gamma_1^{-1})
                \frac{\sigma^2}{\mu_{1,i}}
                +
                \frac{\sigma^4}{\gamma_1 \mu_i^2}
            \right), \\
    \theta_{1,i} \varphi_{1,i}
        =
            \rho
            \cdot
            \sqrt{\frac{\mu_i}{\bmu_{1,i}}}
            \cdot
            \left(
                1
                -
                \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
            \right),
\end{gather*}
so that
\begin{multline*}
    -2 \sqrt{\frac{\mu_i}{\bmu_{1,i}}} \theta_{1,i} \varphi_{1,i}
    +
    \frac{\mu_i}{\bmu_i} \theta_{1,i}^2 \varphi_{1,i}^2 \\
        =
            -
            \rho^2
            \left(
                \frac{\mu_i}{\bmu_{1,i}}
            \right)^2
            \left(
                1
                -
                \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
            \right)
            \left(
                1
                +
                2( 1 + \gamma_1^{-1}) \frac{\sigma^2}{\mu_{1,i}}
                +
                3
                \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
            \right),
\end{multline*}
\[
    \frac{\sigma^2}{\bmu_{1,i}}
    \left(
        \gamma^{-1} \theta_{1,i}^2
        +
        \varphi_{1,i}^2
    \right)
        =
            \rho^2
            \left(
                \frac{\mu_i}{\bmu_{1,i}}
            \right)^2
            \left(
                1
                -
                \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
            \right)
            \left(
                (1 + \gamma_1^{-1})
                \frac{\sigma^2}{\mu_{1,i}}
                +
                2
                \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
            \right),
\]
and
\[
    \frac{1}{\mu_i}
    \frac{\sigma^4}{\gamma \bmu_{1,i}}
        =
            \rho^2
            \left(
                \frac{\mu_i}{\bmu_{1,i}}
            \right)^2
            \left(
                \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
            \right)
            \left(
                1
                +
                (1 + \gamma_1^{-1})
                \frac{\sigma^2}{\mu_{1,i}}
                +
                \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
            \right).
\]
Putting it all together, we get
\begin{align}
    \begin{split}
    \beta_i
        &=
            1
            +
            \rho^2
            \left(
                \frac{\mu_i}{\bmu_{1,i}}
            \right)^2 
            \cdot
            \bigg\{
                \left(
                    1
                    -
                    \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
                \right)
                \left(
                    -
                    1
                    -
                    (1 + \gamma_1^{-1})
                    \frac{\sigma^2}{\mu_{1,i}}
                    -
                    \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
                \right) \\
            &\qquad\qquad\qquad\qquad\qquad+
                \left(
                    \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
                \right)
                \left(
                    1
                    +
                    (1 + \gamma_1^{-1})
                    \frac{\sigma^2}{\mu_{1,i}}
                    +
                    \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
                \right)
            \bigg\}
    \end{split} \notag \\
    &=
        1
        +
        \rho^2
        \left(
            \frac{\mu_i}{\bmu_{1,i}}
        \right)^2
        \left(
            2
            \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
            -
            1
        \right)
        \left(
            1
            +
            (1 + \gamma_1^{-1})
            \frac{\sigma^2}{\mu_{1,i}}
            +
            \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
        \right) \notag \\
    &=
        1
        +
        \frac{2 \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2} - 1}
             {
                1
                +
                (1 + \gamma_1^{-1})
                \frac{\sigma^2}{\mu_{1,i}}
                +
                \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
             } \notag \\
    &=
        \frac{ 3 \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2} 
               +
               (1 + \gamma_1^{-1})
               \frac{\sigma^2}{\mu_{1,i}}
             }
             {
                1
                +
                (1 + \gamma_1^{-1})
                \frac{\sigma^2}{\mu_{1,i}}
                +
                \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
             } \notag \\
    &=
        \frac{
            \frac{\sigma^2}{\gamma_1 \mu_{1,i}^2}
            \left(
                3 \sigma^2
                +
                (\gamma_1 + 1)
                \mu_{1,i}
            \right)
        }
        {
            1
            +
            (1 + \gamma_1^{-1})
            \frac{\sigma^2}{\mu_{1,i}}
            +
            \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2}
        }.
\end{align}
In particular, note that $\beta_i < 1$ when
\(
    2 \frac{\sigma^4}{\gamma_1 \mu_{1,i}^2} - 1 < 0,
\)
or equivalently
\(
    \mu_i > \frac{\sigma^2}{\sqrt{\gamma}} \cdot \sqrt{\frac{2}{\rho}}.
\)
Getting the final expressions for $\beta_i$ and $\eta$ in 
Theorem~\ref{T:bcv-expected-me} is a matter of routine algebra.

\section{Summary and future work}\label{S:bcv-theory-summary}

We have provided an analysis of the first-order behavior of bi-cross-validation.  This analysis has shown that BCV gives a biased estimate of prediction error, with an explicit expression for the bias.  Fortunately, the bias is not too bad when the signal strength is large and the leave-out sizes are small.  Importantly, our analysis gives guidance as to how the leave-out sizes should be chosen.  Our theoretical analysis agrees with the simulations done by Owen \& Perry~\cite{owen2009bi}, who observed that despite bias in prediction error estimates, larger hold-out sizes tend to perform better at estimating $k_\Frob^\ast$.

The form of consistency we give is rather weak since we did not analyze the variance of the BCV prediction error estimate.  As a follow-up, it would be worthwhile to address this limitation.  For now, though, we can get some comfort from empirical observations in \cite{owen2009bi} that the variance of the estimator is not too large.  Indeed, based on these simulations, it is entirely possible that the variance becomes negligible for large $n$ and $p$.

bibliography.tex

\bibliographystyle{plain}
\bibliography{perry-thesis-sources}

cv-unsupervised.tex

\chapter{Cross-validation for unsupervised learning}\label{C:cvul}

The problem unsupervised learning (UL) tries to address is this: given some
data, describe its distribution. Many estimation problems can be cast as
unsupervised learning, including mean and density estimation. However, more commonly unsupervised learning refers to
either clustering or manifold learning. A canonical example is principal
component analysis (PCA). In PCA, we are given some high-dimensional data,
and we look for a lower-dimensional subspace that explains most of the
variation in the data. The lower-dimensional subspace describes the
distribution of the data. In clustering, the estimated cluster centers give us
information about the distribution of the data. The output of every UL method
is a summary statistic designed to convey information about the process which
generated the data.

Many UL problems involve model selection. For example, in principal component
analysis we need to choose how many components to keep. For clustering, we
need to choose the number of clusters in the data. Many manifold learning
techniques require choosing a bandwidth or a kernel. Often in these contexts,
model-selection is done in an ad-hoc manner. Rules of thumb and manual
inspection guide most choices for how many components to keep, how many
clusters are present, and what is an appropriate kernel. Such informal
selection rules can be problematic when different researchers come to different
conclusions about what the right model is. Moreover, even when there is an
obvious ``natural'' model to human eyes, it may be hard to pick out in
computer-automated analysis. For objectivity and efficiency, it is desirable to have a well-specified automatic model selection procedure.

For concreteness, in this chapter we focus on principal components, though
many of the ideas generalize to other methods. We suppose that the data,
$\mX$, is an $n \times p$ matrix generated according to the signal-plus-noise
model $\mX = \sqrt{n} \, \mU \mD \mV^\trans + \mE$. We consider the first term to
be the ``signal'' matrix, and the second term to be ``noise''. Often the
signal term is a low-rank product. Our goal is to estimate this term as
best as possible by truncating the singular value decomposition (SVD) of
$\mX$. Here ``best'' means with respect to the metrics introduced in
Chapter~\ref{C:intrinsic-rank}. We are interested in the model selection
problem where each model is defined by the number of terms we keep from the
SVD of $\mX$.

We would like our model selection procedure to be non-parametric, if possible.
To work in a variety of contexts, the selection procedure cannot assume
Gaussianity or independence across samples. We would like the procedure to be
driven by the empirical distribution of the data. Cross-validation (CV) is a
popular approach to model selection that generally meets these criteria.
Therefore, we seek to adapt CV to our purposes.

CV prescribes dividing a data set into a ``test'' set and a
``train'' set, fitting a model to the training set, and then evaluating the
model's performance on the test set. We repeat the fit/evaluate procedure
multiple times over different test/train partitions, and then average over all
replicates. Traditionally, the partitions are chosen so that each datum occurs
in exactly one test set. As for terminology, for a particular replicate the
test set is commonly referred to as the held-out or left-out set, and likewise
the train set is often called the held-in or left-in set.

Most often, cross-validation is applied in supervised contexts. In supervised
learning (SL) the data consists of a sequence of $(\vx, \vy)$
predictor-response pairs. Broadly construed, the goal of supervised learning
is to describe the conditional distribution of $\vy$ given $\vx$. This is
usually for prediction or classification.  In the supervised context, for a 
particular CV replicate there are four parts of data:
\[
    \left(
    \begin{matrix}
        \mX_\train & \mY_\train \\
        \mX_\test  & \mY_\test
    \end{matrix}
    \right).
\]
Implicit in the description of cross-validation is that the replicates use
$\mX_\test$ to predict $\mY_\test$. So, the held-in data looks like
\[
    \left(
    \begin{matrix}
        \mX_\train & \mY_\train \\
        \mX_\test  & \missing
    \end{matrix}
    \right).
\]
We extrapolate from $\mX_\test$ to predict $\mY_\test$.

It is not immediately obvious how to apply cross-validation to unsupervised
learning.  In unsupervised learning there is no $\mY$; we instead have the 
two-way partition
\[
    \left(
    \begin{matrix}
        \mX_\train \\
        \mX_\test
    \end{matrix}
    \right).
\]
There is nothing to predict!  Renaming $\mX$ to $\mY$ does not make the
problem any better, for then the division becomes:
\[
    \left(
    \begin{matrix}
        \mY_\train \\
        \mY_\test
    \end{matrix}
    \right),
\]
with hold-in
\[
    \left(
    \begin{matrix}
        \mY_\train \\
        \missing
    \end{matrix}
    \right).
\]
Now, there is nothing to extrapolate from to predict $\mY_\test$!  For 
cross-validation to work in unsupervised learning, we need to consider
more general hold-outs.

We look at two different types of hold-outs in this chapter. The first, due to
Wold, is ``speckled'': we leave out random elements of the matrix $\mX$ and
use a missing data algorithm like expectation-maximization (EM) for
prediction. The second type of hold-out is ``blocked''. This type, due to
Gabriel, randomly partitions the columns of $\mX$ into ``predictor'' and
``response'' sets and then performs the SL version of cross-validation.

\section{Assumptions, and notation}\label{S:ucv-assumptions-notation}

We will generally assume we have data $\mX \in \reals^{n \times p}$
generated according to the latent factor model 
\begin{equation}
    \mX = \sqrt{n} \, \mU \mD \mV^\trans + \mE.
\end{equation}
Here, $\mU \in \reals^{n \times k_0}$, $\mV \in \reals^{n \times k_0}$, and
$\mD = \diag( d_1, d_2, \ldots, d_{k_0} )$, with $\mU^\trans \mU = \mI_{k_0}$
$\mV^\trans \mV = \mI_{k_0}$, and $d_1 \geq d_2 \geq \cdots \geq d_{k_0} > 0$. We call $\sqrt{n} \, \mU \mD \mV^\trans$ the
signal part and $\mE$ the noise part. In the spirit of data-driven analysis,
we avoid putting distributional assumptions on $\mU$, $\mD$, $\mV$, and $\mE$.
This makes the terms unidentifiable.  While this indeterminacy can (and 
should!) bother some readers, for now we will plod on.

We denote the SVD of $\mX$ by
\begin{equation}
    \mX = \sqrt{n} \, \mhU \mhD \mhV^\trans, 
\end{equation}
with $\mhU \in \reals^{n \times n \wedge p}$, $\mhV \in \reals^{p \times n
\wedge p}$, and $\mhD = \diag( \hd_1, \hd_2, \ldots, \hd_{n \wedge p})$. Here,
$\mhU^\trans \mhU = \mhV^\trans \mhV = \mI_{n \wedge p}$ and the singular
values are ordered
\(
    \hd_1 \geq \hd_2 \geq \cdots \geq \hd_{n \wedge p} \geq 0.
\)
We set $\mhD(k) = \diag( \hd_1, \hd_2, \ldots, \hd_k, 0, \ldots, 0) \in \reals^{n \wedge p \times n \wedge p}$ so that
\begin{equation}
    \mhX(k) = \sqrt{n} \, \mhU \mhD(k) \mhV^\trans
\end{equation}
is the SVD of $\mX$ truncated to $k$ terms.  Similarly, we define $\mhU(k) \in 
\reals^{n \times k}$ and $\mhV(k) \in \reals^{p \times k}$ to be the first $k$ 
rows of $\mhU$ and $\mhV$, respectively.

We focus on estimating the squared Frobenius model error
\begin{equation}
    \ME(k) = \| \sqrt{n} \, \mU \mD \mV^\trans - \mhX(k) \|_\Frob^2
\end{equation}
or its minimizer,
\begin{equation}
    k^\ast_{\ME} = \argmin_k \ME(k).
\end{equation}
Here, $\| \cdot \|_\Frob^2$ is the sum of squares of the elements.

Another kind of error relevant to cross-validation is \emph{prediction error}.  We let $\mE'$ be a matrix independent of $\mE$ but having the same distribution conditionally on $\mU$, $\mD$, and $\mV^\trans$.  We set 
\(
    \mX' = \sqrt{n} \, \mU \mD \mV^\trans + \mE'
\)
and define the prediction error
\begin{equation}
    \PE(k) = \E \| \mX' - \mhX(k) \|_\Frob^2.
\end{equation}
Likewise, we set
\begin{equation}
    k^\ast_{\PE} = \argmin_k \PE(k).
\end{equation}
If $\mE$ is independent of $\mU$, $\mD$, and $\mV$, then
\begin{align}
    \PE(k) 
        &= \E\| \sqrt{n} \, \mU \mD \mV^\trans - \mhX(k) + \mE' \|_\Frob^2 
        \notag \\
    \begin{split}
        &= \E\| \sqrt{n} \, \mU \mD \mV^\trans - \mhX(k) \| \\
           &\qquad\qquad+ 
           2 \, 
           \E \left[
               \tr \left( 
                   \big( \sqrt{n} \, \mU \mD \mV^\trans - \mhX(k) \big)^\trans 
                   \mE'
               \right)
           \right]
           +
           \E\| \mE' \|_\Frob^2 
    \end{split} \notag \\
        &= \E\big[\ME(k)\big] + \E \| \mE \|_\Frob^2.
\end{align}
The prediction error is thus equal to the sum of the expected model error
and an irreducible error term.

Finally, we should note that our definitions of prediction error and model
error are motivated by the definitions given by
Breiman~\cite{breiman1992little} for cross-validating linear regression.


\section{Cross validation strategies}

In this section we describe the various hold-out strategies for getting a
cross-validation estimate of $\PE(k)$. It is possible to get an estimate of
$\ME(k)$ from the estimate of $\PE(k)$ by subtracting an estimate of the
irreducible error. For now, though, we choose to focus just on estimating
$\PE(k)$.

For performing $K$-fold cross-validation on the matrix $\mX$, we partition its
elements into $K$ hold-out sets. For each of $K$ replicates and for each value
of the rank, $k$, we leave out one of the hold-out sets, fit a $k$-term SVD to
the left-in set, and evaluate its performance on the left-out set. We thus
need to describe the hold-out set, how to fit an SVD to the left-in data, and
how to make a prediction of the left-out data. After a brief discussion of why
the usual (naive) way of doing hold-outs won't work, we survey both speckled
hold-outs (Wold-style), as well as blocked hold-outs (Gabriel-style).

\subsection{Naive hold-outs}

The ordinary hold-out strategy will not work for estimating prediction error.  Suppose we leave out a subset of the rows of $\mX$.  After
permutation, the rows of $\mX$ are partitioned as
\(
    \left(
    \begin{matrix}
        \mX_1 \\
        \mX_2
    \end{matrix}
    \right),
\)
where $\mX_1 \in \reals^{n_1 \times p}$ and $\mX_2 \in \reals^{n_2 \times p}$,
and $n_1 + n_2 = n$.  The only plausible prediction of $\mX_2$ based on truncating the SVD of $\mX_1$ is the following:
\begin{enumerate}
    \item Let $\mX_1 = \sqrt{n} \, \mhU_1 \mhD_1 \mhV_1^\trans$ be the SVD of
        $\mX_1$, with 
        \(
            \mhD_1 
                = 
                \diag( 
                    \hd_1^{(1)}, 
                    \hd_2^{(1)}, 
                    \ldots, 
                    \hd_{n_1 \wedge p}^{(1)}
                ).
        \)
    \item Let 
        \(
            \mhD_1(k)
                =
                \diag(
                    \hd_1^{(1)}\!\!\!, \,
                    \hd_2^{(1)}\!\!\!, 
                    \ldots, 
                    \hd_k^{(1)}\!\!\!, \,
                    0,
                    \ldots,
                    0
                )
        \)
        so that
        $\mhX_1(k) \equiv \sqrt{n} \, \mhU_1 \mhD_1(k) \mhV_1^\trans$ is
        the SVD of $\mX_1$ truncated to $k$ terms.  Similarly, denote
        by $\mhV_1(k) \in \reals^{p \times k}$ the first $k$ columns of
        $\mhV_1$.
    \item Let $(^+)$ denote pseudo-inverse and predict the held out rows as
        \(
            \mhX_2(k) 
                = 
                    \mX_2 \,
                    \mhX_1(k)^\trans
                    \big(
                        \mhX_1(k) \mhX_1(k)^\trans
                    \big)^{+}
                    \mhX_1(k)
                =
                    \mX_2 \, \mhV_1(k) \mhV_1(k)^\trans.
        \)
\end{enumerate}
The problem with this procedure is that $\| \mX_2 - \mhX_2(k) \|_\Frob^2$ 
decreases with $k$ regardless of the true model for $\mX$.  So, it cannot
possibly give us a good estimate of the error from truncating the full
SVD of $\mX$.

A similar situation arrises if we leave out only a subset of the columns of
$\mX$.  To get a reasonable cross-validation estimate, it is therefore 
necessary to consider more-general hold-out sets.

\subsection{Wold hold-outs}

A Wold-style speckled leave-out is perhaps the most obvious attempt at a
more general hold-out. We leave out a subset of the elements of the $\mX$, then
use the left-in elements to predict the rest.

First we need to introduce some more notation. Let $\mathcal{I}$ denote the
set of indices of the elements of $\mX$, so that $\mathcal{I} = \{ (i,j) : 1
\leq i \leq n, 1 \leq j \leq p \}$. For a subset $I \subset \mathcal{I}$, let
$\bar I$ denote its complement $\mathcal{I} \setminus I$. We use the symbol
$\missing$ to denote a missing value, and let 
\(
    \reals_\missing = \reals \cup \{ \missing \}.
\)
For $I \subset \mathcal{I}$, we define the matrix $\mX_I \in
\reals_\missing^{n \times p}$ with elements
\begin{equation}
    X_{I,ij}
    =
    \begin{cases}
        X_{ij}     &\text{if $(i,j) \in I$,} \\
        \missing   &\text{otherwise.}
    \end{cases}
\end{equation}
Similarly $\mX_{\bar I}$, has elements
\begin{equation}
    X_{\bar I,ij}
    =
    \begin{cases}
        X_{ij}     &\text{if $(i,j) \notin I$,} \\
        \missing   &\text{otherwise.}
    \end{cases}
\end{equation}
Finally, for $\mA \in \reals_\missing^{n \times p}$, we define
\begin{equation}
    \| \mA \|_{\Frob,I}^2
        =
        \sum_{(i,j) \in I} A_{ij}^2.
\end{equation}
This notation allows us to describe matrices with missing entries.

A Wold-style speckled hold-out is an unstructured random subset $I \subset \mathcal{I}$.  The 
held-in data is the matrix $\mX_{\bar I}$ and the held-out data is the matrix 
$\mX_I$.  We use a missing value SVD algorithm to fit a $k$-term SVD to 
$\mX_{\bar I}$.  This gives us an approximation $\mU_k \mD_k \mV_k^\trans$, 
where $\mU_k \in \reals^{n \times k}$ and $\mV_k \in \reals^{p \times k}$ have 
orthonormal columns and $\mD \in \reals^{k \times k}$ is diagonal.  We 
evaluate the SVD on the held-out set by 
\(
    \| \mU_k \mD_k \mV_k^\trans - \mX_I \|_{\Frob,I}^2.
\)
Aside from the algorithm for getting $\mU_k \mD_k \mV_k^\trans$ from 
$\mX_{\bar I}$, this is a full description of the CV replicate.

When Wold introduced this form of hold-out in 1978~\cite{wold1978cross}, he
suggested using an algorithm called nonlinear iterative partial least squares
(NIPALS) to fit an SVD to the held-in data. This algorithm, attributed to 
Fisher~and~Mackenzie~\cite{fisher1923scv} and rediscovered by Wold and Lyttkens~\cite{wold1969nonlinear}, never
seems to have gained much prominence. We suggest instead using an
expectation-maximization (EM) as is consistent with current mainstream
practice in Statistics. Either way, there are some subtle issues in taking the
SVD of a matrix with missing values. We explore these issues further in
Section~\ref{S:missing-values-svds}, and present a complete algorithm for
estimating the factors $\mU_k \mD_k \mV_k^\trans$.

\subsection{Gabriel hold-outs}

A Gabriel-style hold-out works by transforming the unsupervised learning
problem into a supervised one. We take a subset of the columns of $\mX$ and
denote them as the \emph{response} columns; the rest are denoted
\emph{predictor} columns. Then we take a subset of the rows and consider them
as \emph{test} rows; the rest are \emph{train} rows. This partitions the
elements of $\mX$ into four blocks. With permutation matrices $\mP$ and $\mQ$,
we can write
\begin{equation}
    \mP^\trans \mX \mQ
        =
        \left(
        \begin{matrix}
           \mX_{11} & \mX_{12} \\
           \mX_{21} & \mX_{22}
        \end{matrix}
        \right).
\end{equation}
Here, $\mX_{11}$ consists of the train-predictor block, $\mX_{12}$ is the train-response block, $\mX_{21}$ is the test-predictor block, and $\mX_{22}$ is the test-response block.  It is beneficial to think of the blocks as
\[
    \left(
    \begin{matrix}
       \mX_{11} & \mX_{12} \\
       \mX_{21} & \mX_{22}
    \end{matrix}
    \right)
        =
        \left(
        \begin{matrix}
           \mX_\train & \mY_\train \\
           \mX_\test  & \mY_\test 
        \end{matrix}
        \right).
\]
A Gabriel-style replicate has hold-in
\[
\left(
    \begin{matrix}
       \mX_{11} & \mX_{12} \\
       \mX_{21} & \missing
    \end{matrix}
    \right)
\]
and hold-out $\mX_{22}$.

To fit a model to the hold-in set, we take the SVD of $\mX_{11}$ and fit a 
regression function from the predictor columns to the response columns.  
Normally, the regression is ordinary least squares regression from the 
principal component loadings to the response columns.  Then, to evaluate the 
function on the hold-out set, we apply the estimated regression function to 
$\mX_{21}$ to get a prediction $\mhX_{22}$.

In precise terms, suppose that there are $p_1$ predictor columns, $p_2$ response columns, $n_1$ train rows, and $n_2$ test rows.  Then $\mX_{11} \in \reals^{n_1 \times p_1}$ and $\mX_{22} \in \reals^{n_2 \times p_2}$ with
$p_1 + p_2 = p$ and $n_1 + n_2 = n$.  First we fit an SVD to the train-predictor block 
\begin{equation*}
    \mX_{11} = \sqrt{n} \, \mhU_1 \mhD_1 \mhV_1^\trans.  
\end{equation*}
Then, we truncate the SVD to $k$ terms as 
$\sqrt{n} \, \mhU_1(k) \mhD_1(k) \mhV_1^\trans(k)$ in the same way as is discussed 
in Section~\ref{S:ucv-assumptions-notation}.  This defines a projection 
$\mhV_1(k)$ and principal component scores 
\begin{equation*}
    \mhZ_1(k) = \mX_{11} \mhV_1(k) = \sqrt{n} \, \mhU_1(k) \mhD_1(k).
\end{equation*}
Similarly, we can get the principal components scores for the test-predictor block as
\begin{equation*}
    \mhZ_2(k) = \mX_{21} \mhV_1(k).
\end{equation*}
Next, we model the response columns as linear functions of the principal component scores.  We fit the model
\(
    \mX_{12} \approx \mhZ_1(k) \mB
\)
with
\begin{equation*}
    \mhB 
        =
        \big( \mhZ_1(k)^\trans \mhZ_1(k) \big)^{+} \mhZ_1(k)^\trans \mX_{12}
        =
        \frac{1}{\sqrt{n}} \,
        \mhD_1(k)^{+} \, \mhU_1(k)^\trans \mX_{12}.
\end{equation*}
Finally, we apply the model the test rows to get a prediction for the test-response block
\begin{equation}
    \mhX_{22}
        =
        \mhZ_2(k) \mhB
        =
        \mX_{21} 
        \left(
            \frac{1}{\sqrt{n}} \,
            \mhV_1(k) \mhD_1(k)^{+} \, \mhU_1(k)^\trans 
        \right)
        \mX_{12}.
\end{equation}
This gives a complete description of the CV replicate.

\begin{remark}
Typically for Gabriel-style CV, we have two partitions, one for the rows
and one for the columns.  If the rows are partitioned into $K$ sets and
the columns are partitioned into $L$ sets, then we average the estimated
prediction error over all $K L$ possible hold-out sets.  This corresponds
to $\frac{n}{n_2} \approx K$ and $\frac{p}{p_2} \approx L$.  In this situation, we say that we are performing $(K,L)$-fold Gabriel-style cross-validation.
\end{remark}

We can get some intuition for why Gabriel-style replicates work by expressing 
the latent factor decomposition $\mX = \sqrt{n} \,\mU \mD \mV^\trans + \mE$ in 
block form.  We let
\(
    \mP
    =
    \left(
    \begin{matrix}
        \mP_1 &
        \mP_2
    \end{matrix}
    \right)
\)
and
\(
    \mQ
    =
    \left(
    \begin{matrix}
        \mQ_1 &
        \mQ_2
    \end{matrix}
    \right)
\)
be the block-decompositions of the row and column permutations so that
$\mX_{ij} = \mP_i^\trans \mX \mQ_j$.  We define $\mU_i = \mP_i^\trans \mU$,
$\mV_j = \mQ_j^\trans \mV$, and $\mE_{ij} = \mP_i^\trans \mE \mQ_j$ so that
\begin{align*}
    \mP^\trans \mX \mQ
        &=
            \left(
            \begin{matrix}
                \mP_1^\trans \\
                \mP_2^\trans
            \end{matrix}
            \right)
            \left(
                \sqrt{n} \, \mU \mD \mV^\trans + \mE
            \right)
            \left(
            \begin{matrix}
                \mQ_1 & \mQ_2
            \end{matrix}
            \right) \\
        &=
            \sqrt{n}
            \left(
            \begin{matrix}
                \mU_1 \mD \mV_1^\trans
                    & \mU_1 \mD \mV_2^\trans  \\
                \mU_2 \mD \mV_1^\trans
                    & \mU_2 \mD \mV_2^\trans
            \end{matrix}
            \right)
            +
            \left(
            \begin{matrix}
                \mE_{11} & \mE_{12} \\
                \mE_{21} & \mE_{22}
            \end{matrix}
            \right).
\end{align*}
Thus, all four blocks have the same low-rank structure.  If the noise is small 
then $\mX_{22} \approx \sqrt{n} \, \mU_2 \mD \mV_2^\trans$ and
\[
    \mhX_{22} 
        \approx 
            \sqrt{n} \,
                \mU_2 \mD 
                \big( \mV_1^\trans \mhV_1(k) \big) \,
                \mhD(k)^{+} \,
                \big( \mhU_1(k)^\trans \mU_1 \big) \,
                \mD \mV_2^\trans
\]
If the noise is exactly zero, $k = k_0$, and $\rank(\mX_{22}) = \rank(\mX)$, 
then Owen and Perry~\cite{owen2009bi} show that $\mhX_{22} = \mX_{22}$.  For 
other types of noise, the next chapter proves a more general consistency 
result.

\subsection{Rotated cross-validation}\label{SS:rotated-cv}

When the underlying signal $\mU \mD \mV^\trans$ is sparse, it's possible that we will miss it in the training set.  For example, if 
\(
    \mU 
    =
    \left(
    \begin{matrix}
        1 & 0 & 0 & 0
    \end{matrix}
    \right)^\trans
\)
and
\(
    \mV
    =
    \left(
    \begin{matrix}
        1 & 0 & 0 & 0 & 0
    \end{matrix}
    \right)^\trans
\),
then
\[
    \mU \mD \mV^\trans
    =
    \left(
    \begin{matrix}
        d_{1} & 0 & 0 & 0 & 0 \\
        0     & 0 & 0 & 0 & 0 \\
        0     & 0 & 0 & 0 & 0 \\
        0     & 0 & 0 & 0 & 0 
    \end{matrix}
    \right).
\]
Either the test set will observe this factor, or the train set, but not both.  Only the set that contains $X_{11}$ will be affected by the factor.

One way to adjust for sparse factors is to randomly rotate the rows and
columns of $\mX$ before performing the cross-validation. We generate $\mP \in
\reals^{n \times n}$ and $\mQ \in \reals^{p \times p}$, uniformly random
orthogonal matrices, by employing Algorithm~\ref{A:random-steifel}. Then, we
set $\mtX = \mP \mX \mQ^\trans$ and perform ordinary cross-validation on
$\mtX$. Regardless of the factor structure in $\mX$, the signal part of $\mtX$
will be uniformly spread across all elements of the matrix. We call this
procedure \emph{rotated cross-validation}, abbreviated RCV.  

RCV is similar in spirit to the generalized cross-validation (GCV) described
in Craven \& Wahba~\cite{craven1978smoothing} and Golub et
al.~\cite{golub1979generalized}. GCV is an orthogonally-invariant way of
performing leave-one-out cross validation for regression. The difference is
that GCV rotates to a specific non-random configuration of the data that gives
equal weight to all observations in the rotated space, while RCV rotates to a 
random configuration that gives equal weight in expectation.


\section{Missing-value SVDs}\label{S:missing-values-svds}

To perform a Wold-style cross-validation we need to be able to compute the SVD of a matrix with missing entries.  This is a difficult problem.  First of all, the problem is not very well defined.  If $\mA$ is a matrix with missing entries, there are potentially many different ways of factoring $\mA$ as an SVD-like product.  Often, one attempts to force uniqueness by finding the complete matrix $\mA' \in \reals^{n,p}$ of minimum rank such that $A'_{ij} = A_{ij}$ for all non-missing elements of $\mA$.  Even then, $\mA'$ may not be unique.  Take
\[
    \mA
    =
    \left(
    \begin{matrix}
        1        & \missing \\
        \missing & \missing
    \end{matrix}
    \right).
\]
Then
\[
    \left(
    \begin{matrix}
        1 & 0 \\
        0 & 0
    \end{matrix}
    \right),
    \quad
    \left(
    \begin{matrix}
        1 & 1 \\
        0 & 0
    \end{matrix}
    \right),
    \quad
    \left(
    \begin{matrix}
        1 & 0 \\
        1 & 0
    \end{matrix}
    \right),
    \quad
    \text{and}
    \quad
    \left(
    \begin{matrix}
        1 & 1 \\
        1 & 1
    \end{matrix}
    \right)
\]
are all rank-$1$ matrices that agree with $\mA$ on its non-missing entries.  We might discriminate between these by picking the matrix with minimum Frobenius norm.  In this case, the matrix
\[
    \left(
    \begin{matrix}
        1 & \missing \\
        \missing & 1
    \end{matrix}
    \right)
\]
presents an interesting problem.  We can either complete it as
\[
    \left(
    \begin{matrix}
        1 & 1 \\
        1 & 1
    \end{matrix}
    \right)
    \quad
    \text{or}
    \quad
    \left(
    \begin{matrix}
        1 & 0 \\
        0 & 1
    \end{matrix}
    \right).
\]
The first option has rank $1$ and Frobenius norm $2$.  The second option
has higher rank, $2$, but lower Frobenius norm, $\sqrt{2}$.  Which criterion 
is more important?  It it not clear what the ``right'' SVD of $\mA$ is.

We can alleviate the problem by considering a sequence of SVDs rather than a 
single one.  For each $k = 0, 1, 2, \ldots$, define $\mA_k'$ to the 
rank-$k$ matrix of minimum Frobenius norm that agrees with $\mA$ on its 
non-missing elements.  If no such matrix exists, let $I \subset \mathcal{I}$ 
be indices of the non-missing elements of $\mA$ and define the candidate
sets
\begin{align*}
    \mathcal{A}_k 
        &= \{ \mA_k \in \reals^{n \times p} : \rank(\mA_k) = k \} \\
    \mathcal{C}_k
        &= \{ \mA_k \in  \mathcal{A}_k 
                : \| \mA - \mA_k \|_{\Frob,I}
                  =
                  \min_{\mB_k \in \mathcal{A}_k} 
                      \| \mA - \mB_k \|_{\Frob,I} \}.
\end{align*}
Lastly, put
\[
    \mA'_k
        = \argmin_{\mA_k \in \mathcal{C}_k}
            \| \mA_k \|_{\Frob,I}.
\]
We then define the rank-$k$ SVD of $\mA$ to be the equal to the SVD of 
$\mA'_k$.  

There are still some problems with these SVDs. First of all, in general there
may be no relationship between $\mA_k'$ and $\mA_{k+1}'$; the two matrices may
be completely different and have completely different SVDs. We lose the
nesting property of ordinary SVDs, where the rank-$k$ SVD is contained in the
rank-$(k+1)$ SVD. Secondly, although it seems plausible, we do not have any
guarantees that $\mA_k'$ is unique. Situations may arise where two different
rank-$k$ matrices have the same norms and the same residual norms on the
non-missing elements of $\mA$. Finally, finding $\mA'_k$ is a non-convex
problem. The function $\| \mA - \mA_k \|_{\Frob,I}$ can have more than one
local maximum.  We need to be aware of these deficiencies.

Rather than get too deep into the missing-value SVD rabbit-hole, we choose
instead to live with an approximation. We acknowledge that computing the best
rank-$k$ approximation as defined above is computationally infeasible for
large $n$ and $p$. Instead of proving theorems about the global optimum, we
focus on an algorithm for computing a local solution.

We use an EM-algorithm to estimate $\mA'_k$. The inputs to the algorithm are
$k$, a non-negative integer rank, and $\mA \in \reals_\missing^{n \times p}$,
a matrix with missing values. The output is $\mA_k'$, a rank-$k$ approximation
of $\mA$. The algorithm proceeds by iteratively estimating the missing values
of $\mA$ by the values from the first $k$ terms of the SVD of the completed
matrix. We give detailed pseudocode for the procedure as
Algorithm~\ref{A:em-svd}. This is essentially the same algorithm as the
SVDimpute algorithm given in Troyanksaya et al.~\cite{troyanskaya2001missing},
except that we use a different convergence criterion. SVDimpute stops when
successive estimates of the missing values of $\mA$ differ by less than ``the
empirically determined threshold of $0.01$.'' We instead stop when relative
difference of the residual sum of squares (RSS) between the non-missing
entries and the rank-$k$ SVD is small (usually $1.0 \times 10^{-4}$ or less).
Our reason for using a different convergence rule is that the analysis of the
EM algorithm in Dempster et al.~\cite{dempster1977maximum} shows that the RSS
decreases with each iteration, but makes no assurances about the missing
values converging. Regardless of which convergence criterion is used, the
algorithm is very easy to implement.

\begin{algorithm}
    \caption{Rank-$k$ SVD approximation with missing values}\label{A:em-svd}
    
    \begin{enumerate}
        \item Let $I = \{ (i,j) : A_{ij} \neq \missing \}$.
        \item For $1 \leq j \leq p$ let $\mu_j$ be the mean of the
            non-missing values in column $j$ of $\mA$, or $0$ if all of
            the entries in column $j$ are missing.
        \item Define $\mA^{(0)} \in \reals^{n \times p}$ by
            \[
                A^{(0)}_{ij}
                    =
                    \begin{cases}
                        A_{ij} &\text{if $(i,j) \in I$,} \\
                        \mu_j  &\text{otherwise.}
                    \end{cases}
            \]
        \item Initialize the iteration count $N \gets 0$.
        \item (\textsc{M-Step}) \label{em-svd-m-step} Let 
            \[
                \mA^{(N)} = 
                    \sum_{i=1}^{n \wedge p}
                        d^{(N)}_i \vu^{(N)}_i \vv^{(N)\trans}_i
            \]
            be the SVD of $\mA^{(N)}$ and let $\mA'^{(N)}_k$ be the SVD 
            truncated to $k$ terms, so that
            \[
                \mA'^{(N)}_k 
                    = 
                        \sum_{i=1}^{k}
                            d^{(N)}_i \vu^{(N)}_i \vv^{(N)\trans}_i.
            \]
        \item (\textsc{E-Step}) Define $\mA^{(N+1)} \in \reals^{n \times p}$ 
            as
            \[
                A^{(N+1)}_{ij}
                    =
                    \begin{cases}
                        A_{ij}           &\text{if $(i,j) \in I$,} \\
                        A'^{(N)}_{k,ij}  &\text{otherwise.}
                    \end{cases}
            \]
        \item Set 
            \[
                \text{RSS}^{(N)} = \| \mA - \mA'^{(N)}_k \|^2_{\Frob,I}.
            \]
            If $|\text{RSS}^{(N)} - \text{RSS}^{(N-1)}|$ is small, declare 
            convergence and output $\mA'^{(N)}_k$ as $\mA'_k$.  
            Otherwise, increment $N \gets N + 1$ and go to
            Step~\ref{em-svd-m-step}.
    \end{enumerate}
\end{algorithm}

\clearpage

\section{Simulations}

We performed two sets of simulations to gauge the performance of Gabriel- and Wold-style cross-validation.  In the first set of simulations, we compare estimated prediction error with true prediction error.  In the second set, we evaluate the methods' abilities to estimate $k_{\PE}^\ast$, the optimal rank.

Each simulation generates a data matrix $\mX$ as 
\(
    \mX = \sqrt{n} \, \mU \mD \mV^\trans + \mE.
\)
We fix the dimensions and number of generating signals as $n = 100$, $p = 50$,
and $k_0 = 6$.  For ``weak'' factors we set 
\(
    \mD 
    = 
    \mD_\text{weak}
    =
    \diag \left(
        10, 9, 8, 7, 6, 5
    \right)
\)
and for ``strong'' factors, we set
\(
    \mD
    =
    \mD_\text{strong}
    =
    \sqrt{n} \mD_\text{weak}.
\)
We generate $\mU$, $\mV$, and $\mE$ independently of each other.  To avoid 
ambiguity in what constitutes ``signal'' and what constitutes ``noise'', we 
ensure that the elements of $\mE$ are uncorrelated with each other.

We consider two types of factors.  For ``Gaussian'' factors, we put the elements of $\mU$ distributed iid with
$U_{11} \sim \Normal\big( 0, \, \frac{1}{n} \big)$ and the elements of
$\mV$ iid with $V_{11} \sim \Normal\big( 0, \, \frac{1}{p} \big)$, also independent of $\mU$.  For ``Sparse'' factors we use sparsity parameter $s = 10\%$ and set
\[
    \Prob\{ U_{11} = 0 \} = 1 - s,
\]
\[
    \Prob\Big\{ U_{11} = -\frac{1}{\sqrt{s \, n}} \Big\}
    =
    \Prob\Big\{ U_{11} = +\frac{1}{\sqrt{s \, n}} \Big\}    
    = 
    \frac{s}{2}.
\]
Similarly, we put
\[
    \Prob\{ V_{11} = 0 \} = 1 - s,
\]
\[
    \Prob\Big\{ V_{11} = -\frac{1}{\sqrt{s \, p}} \Big\}
    =
    \Prob\Big\{ V_{11} = +\frac{1}{\sqrt{s \, p}} \Big\}    
    = 
    \frac{s}{2}.
\]
The scalings in both cases are chosen so that $\E [\mU^\trans \mU] = \E 
[\mV^\trans \mV] = \mI_{k_0}$.  Gaussian factors are uniformly spread out in the observations and variables, while sparse factors are only observable in a small percentage of the matrix entries (about $1\%$).

We use three types of noise.  For ``white'' noise, we generate the elements of $\mE$ iid with $E_{11} \sim \Normal\left( 0, \, 1\right)$.  For ``heavy'' noise, we use iid elements with $E_{11} \sim \sigma_\nu^{-1} \, t_\nu$, and
$\nu = 3$.  Here $t_\nu$ is a $t$ random variable with $\nu$ degrees of freedom, and $\sigma_\nu = \sqrt{\nu/(\nu-2)}$ is chosen so that 
$\E [ E_{11}^2 ] = 1$.  Heavy noise is so-called because it has a heavy tail.  Lastly, for ``colored'' noise, we first generate
$\sigma^2_1, \ldots, \sigma^2_n \sim \text{Inverse-$\chi^2$}(\nu_1)$ and
$\tau^2_1, \ldots, \tau^2_p \sim \text{Inverse-$\chi^2$}(\nu_2)$ independently, with $\nu_1 = \nu_2 = 3$.  Then we generate the elements of $\mE$ independently as
\(
    E_{ij} \sim 
        c_{\nu_1, \nu_2}^{-1} \, \cdot \Normal( 0, \, \sigma^2_i + \tau^2_j),
\)
where
\(
    c_{\nu_1, \nu_2}
        =
        \sqrt{1/(\nu_1 - 2) + 1/(\nu_2 - 2)}
\).
Again, $c_{\nu_1, \nu_2}$ is chosen so that $\E [E_{ij}^2] = 1$.  Colored 
noise simulates heteroscedasticity.  All three types of noise are plausible 
for real-world data.

Obviously, this set of simulations comes with a number of caveats.  We are
only looking at two choices for the signal strengths, one choice of $n$ and $p$, and a single hold-out size.  Moreover, this example uses relatively small $n$ and $p$, potentially too small for consistency asymptotics to kick in.  Despite these deficiencies, the simulations still convey substantial information about the behavior of the procedures under consideration.

\subsection{Prediction error estimation}

Our goal with the first simulation was to get intuition for the behavior of
Gabriel- and Wold-style cross validation as prediction error estimators. We
generated random data of the form $\mX = \sqrt{n} \mU \mD \mV^\trans + \mE$,
described above. With $\mhX(k)$ being the SVD of $\mX$ truncated to $k$ terms,
the (normalized) true prediction error is given by
\[ 
    \PE(k) = \| \sqrt{n} \, \mU \mD \mV^\trans - \mhX(k) \|_\Frob^2 + 1. 
\] 
Cross-validation gives us an estimate $\widehat{\PE}(k)$ of the prediction 
error curve.  We wanted to see how $\widehat{\PE}(k)$ compares to $\PE(k)$.

Figures~\ref{F:cvsvd-pe-strong}~and~\ref{F:cvsvd-pe-weak} show the true and
estimated prediction error curves from $(2,2)$-fold Gabriel CV and $5$-fold
Wold CV, along with their RCV variants. The plots only show one set of curves
for each factor and noise instance, but other replicates showed similar
behavior.  

For most of the simulations, the cross-validation estimates of prediction
error are generally conservative. The only exception is with weak factors and
colored noise, perhaps because of the ambiguity in what constitutes ``signal''
and what constitutes ``noise'' in this simulation. This global upward bias
agrees with previous studies of cross-validation 
(e.g.~\cite{breiman1992little} and \cite{breiman1992submodel}).  The RCV versions of the methods generally brought down the bias.  Some authors have observed a downward bias at the minimizer of $\widehat{\PE}(k)$ for ordinary cross-validation (\cite{breiman1984cart}, \cite{tibshirani2009}).  This bias does not appear to be present here.

A striking difference between Wold- and Gabriel-style CV is their behavior for $k$ greater than $k^\ast_{\PE}$.  Gabriel-style CV does a better job at estimating the true $\PE(k)$, which is relatively flat.  Wold-style CV, on the other hand, increases steeply for $k$ past the minimizer.  In some situations,
the Wold-style behavior is more desirable, but as the heavy-noise examples in Figure~\ref{F:cvsvd-pe-weak} illustrate, the steep increase is not always in the right place.  Gabriel-style cross-validation is better at conveying ambiguity when the underlying dimensionality of the data is unclear.

\begin{figure}[ht]
    \centering
    \begin{minipage}{0.49\textwidth}
        \begin{center}
            \includegraphics[scale=0.66]{cvsvd-strong-gauss}
        \end{center}
    \end{minipage}
    \begin{minipage}{0.49\textwidth}
        \begin{center}
            \includegraphics[scale=0.66]{cvsvd-strong-sparse}
        \end{center}
    \end{minipage}
    \caption{
        \captiontitle{Cross-validation with strong factors}
        Estimated prediction error curves for Gabriel- and Wold-style
        cross-validation, both original and rotated (RCV) versions, with
        strong factors in the data.  
        The true prediction error is shown in black, the CV curves are red, 
        and the RCV curves are blue.  Error bars are computed from the 
        CV replicates.  Despite their upward bias, the methods do
        well at estimating $\PE(k)$ for $k \leq k^\ast_{\PE}$.
    }\label{F:cvsvd-pe-strong}
\end{figure}

\begin{figure}[ht]
    \centering
    \begin{minipage}{0.49\textwidth}
        \begin{center}
            \includegraphics[scale=0.66]{cvsvd-weak-gauss}
        \end{center}
    \end{minipage}
    \begin{minipage}{0.49\textwidth}
        \begin{center}
            \includegraphics[scale=0.66]{cvsvd-weak-sparse}
        \end{center}
    \end{minipage}
    \caption{
        \captiontitle{Cross-validation with weak factors}
        Estimated prediction error curves for Gabriel- and Wold-style
        cross-validation, both original and rotated (RCV) versions, with
        weak factors in the data.  As in Figure~\ref{F:cvsvd-pe-strong} the
        true prediction error is shown in black, the CV curves are red, and
        the RCV curves are blue.  Error bars give are computed from the 
        CV replicates.  The methods have a harder time estimating $\PE(k)$
        and its minimizer.
    }\label{F:cvsvd-pe-weak}
\end{figure}

\clearpage 

\subsection{Rank estimation}

In the next simulation, we see how well cross-validation works at estimating
the optimal rank, especially as compared to other rank-selection methods.  We 
generate data in the same manner as before, and record how far off the 
minimizer of $\widehat{\PE}(k)$ is from the minimizer of $\PE(k)$.

We compared the four CV-based rank estimation methods with seven other 
methods.  They are as follows:
\begin{itemize}
    \item AIC: Rao \& Edelman's AIC-based estimator 
        \cite{rao2008sample}.
    \item BIC$_1$, BIC$_2$, and BIC$_3$: Bai \& Ng's BIC-based estimators 
        \cite{bai2002dnf}.
    \item $F$: Faber \& Kowalski's modification of 
        Malinowski's  $F$-test, with a significance level of $0.05$  
        \cite{faber1997critical,malinowski1989statisticalf}.
    \item MDL: Wax \& Kailaith's estimator based on the minimum description 
        length principle
        \cite{wax1985detection} .
    \item UIP: Kritchman \& Nadler's estimator based on Roy's  
        union-intersection principle and their background noise estimator,      
        with a significance level of $0.001$
         \cite{kritchman2008determining,roy1953heuristic}.
\end{itemize}
Kritchman and Nadler \cite{kritchman2008determining} give concise descriptions 
of four of the estimators.  The other estimators, Bai and Ng's BICs, are 
defined as the minimizers of
\begin{subequations}
\begin{align}
    \text{BIC}_1(k)
        &= \log \| \mX - \mhX(k) \|_\Frob^2
           + k \, \frac{n + p}{n \, p} \log \frac{n \, p}{n + p}, \\
    \text{BIC}_2(k)
        &= \log \| \mX - \mhX(k) \|_\Frob^2
           + k \, \frac{n + p}{n \, p} \log C_{n,p}, \\
    \text{BIC}_3(k)
        &= \log \| \mX - \mhX(k) \|_\Frob^2
           + k \, \frac{\log C_{n,p}^2}{C_{n,p}},
\end{align}
\end{subequations}
where $C_{n,p} = \min( \sqrt{n}, \sqrt{p})$.

Tables~\ref{T:rank-est-strong-gauss}--\ref{T:rank-est-weak-sparse} summarize
the results of 100 replicates.  For the strong factors in white noise, almost
all of the methods correctly estimate the true $\PE$-minimizing rank.  When the noise is non-white, Wold-style CV seems to be the clear winner.  However, as Figure~\ref{F:cvsvd-pe-strong} demonstrates, there is not much Frobenius loss penalty for slightly overestimating the rank.  Therefore, Tables~\ref{T:rank-est-strong-gauss}~and~\ref{T:rank-est-strong-sparse} may be exaggerating the advantage of Wold-style CV.

For the weak factors in white noise, the AIC, $F$ and UIP methods fare well, and the performance of the cross-validation-based methods is mediocre.  For non-white noise and weak factors, none of the methods perform very well.  This is probably due to the inherent ambiguity between what constitutes ``signal'' and what constitutes ``noise'' in these simulations.

\begin{table}
    \tiny
    \centering    
    \begin{tabular}{lrrrrrrrrrrrrrrrr}
        \toprule
        &\multicolumn{15}{c}{\scriptsize{Estimated Rank}} \\
        \cmidrule{2-17}
        \scriptsize{Method}
            & $-7$ & $-6$ & $-5$ & $-4$ & $-3$ & $-2$ & $-1$ 
            & $\phantom{+}0$ 
            & $+1$ & $+2$ & $+3$ & $+4$ & $+5$ & $+6$ & $+7$ & $> 7$ \\
        \midrule
        \\
        &\multicolumn{16}{l}{\scriptsize{\textit{White Noise}}} \\
 CV-Gabriel &  &  &  &  &  &  &  &  99 &  1 &  &  &  &  &  &  & \\ 
 RCV-Gabriel &  &  &  &  &  &  &  &  97 &  2 &  1 &  &  &  &  &  & \\ 
 CV-Wold &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
 RCV-Wold &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
 AIC &  &  &  &  &  &  &  &  99 &  1 &  &  &  &  &  &  & \\ 
 BIC$_1$ &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
 BIC$_2$ &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
 BIC$_3$ &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
 $F$ &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
 MDL &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
 UIP &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
         \\
        &\multicolumn{16}{l}{\scriptsize{\textit{Colored Noise}}} \\
CV-Gabriel &  &  &  &  &  &  &  &  32 &  42 &  18 &  5 &  2 &  1 &  &  & \\ 
 RCV-Gabriel &  &  &  &  &  &  &  &  6 &  7 &  18 &  13 &  15 &  26 &  9 &  4 &  2\\ 
 CV-Wold &  &  &  &  &  &  &  &  97 &  3 &  &  &  &  &  &  & \\ 
 RCV-Wold &  &  &  &  &  &  &  &  22 &  39 &  29 &  7 &  2 &  1 &  &  & \\ 
 AIC &  &  &  &  &  &  &  &  &  &  &  &  &  2 &  9 &  19 &  70\\ 
 BIC$_1$ &  &  &  &  &  &  &  &  14 &  26 &  31 &  20 &  4 &  4 &  1 &  & \\ 
 BIC$_2$ &  &  &  &  &  &  &  &  23 &  41 &  24 &  9 &  1 &  2 &  &  & \\ 
 BIC$_3$ &  &  &  &  &  &  &  &  &  1 &  1 &  3 &  4 &  5 &  3 &  4 &  79\\ 
 $F$ &  &  &  &  &  &  &  &  &  4 &  11 &  29 &  27 &  15 &  12 &  1 &  1\\ 
 MDL &  &  &  &  &  &  &  &  13 &  24 &  36 &  20 &  3 &  3 &  1 &  & \\ 
 UIP &  &  &  &  &  &  &  &  &  1 &  5 &  14 &  28 &  23 &  19 &  9 &  1\\ 
        \\
        &\multicolumn{16}{l}{\scriptsize{\textit{Heavy Noise}}} \\
CV-Gabriel &  &  &  &  &  &  &  &  61 &  35 &  4 &  &  &  &  &  & \\ 
 RCV-Gabriel &  &  &  &  &  &  &  &  42 &  27 &  12 &  14 &  5 &  &  &  & \\ 
 CV-Wold &  &  &  &  &  &  &  &  99 &  1 &  &  &  &  &  &  & \\ 
 RCV-Wold &  &  &  &  &  &  &  &  68 &  26 &  5 &  1 &  &  &  &  & \\ 
 AIC &  &  &  &  &  &  &  &  3 &  20 &  22 &  32 &  18 &  4 &  1 &  & \\ 
 BIC$_1$ &  &  &  &  &  &  &  &  65 &  27 &  7 &  1 &  &  &  &  & \\ 
 BIC$_2$ &  &  &  &  &  &  &  &  70 &  26 &  3 &  1 &  &  &  &  & \\ 
 BIC$_3$ &  &  &  &  &  &  &  &  32 &  27 &  20 &  13 &  4 &  4 &  &  & \\ 
 $F$ &  &  &  &  &  &  &  &  38 &  29 &  27 &  6 &  &  &  &  & \\ 
 MDL &  &  &  &  &  &  &  &  64 &  28 &  7 &  1 &  &  &  &  & \\ 
 UIP &  &  &  &  &  &  &  &  18 &  35 &  27 &  18 &  2 &  &  &  & \\ 
        \bottomrule
    \end{tabular}
    \caption{
        \captiontitle{Rank estimation with strong Gaussian factors}
        Difference between the estimated rank and the true minimizer of 
        $\PE(k)$ for 100 replicates of strong Gaussian factors with various
        types of noise.
    }\label{T:rank-est-strong-gauss}
\end{table}

\begin{table}
    \tiny
    \centering    
    \begin{tabular}{lrrrrrrrrrrrrrrrr}
        \toprule
        &\multicolumn{15}{c}{\scriptsize{Estimated Rank}} \\
        \cmidrule{2-17}
        \scriptsize{Method}
            & $-7$ & $-6$ & $-5$ & $-4$ & $-3$ & $-2$ & $-1$ 
            & $\phantom{+}0$ 
            & $+1$ & $+2$ & $+3$ & $+4$ & $+5$ & $+6$ & $+7$ & $> 7$ \\
        \midrule
        \\
        &\multicolumn{16}{l}{\scriptsize{\textit{White Noise}}} \\
CV-Gabriel &  &  &  &  &  &  &  8 &  73 &  11 &  7 &  1 &  &  &  &  & \\ 
 RCV-Gabriel &  &  &  &  &  &  &  &  98 &  2 &  &  &  &  &  &  & \\ 
 CV-Wold &  &  &  &  &  &  1 &  8 &  91 &  &  &  &  &  &  &  & \\ 
 RCV-Wold &  &  &  &  &  &  &  1 &  99 &  &  &  &  &  &  &  & \\ 
 AIC &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
 BIC$_1$ &  &  &  &  &  &  &  1 &  99 &  &  &  &  &  &  &  & \\ 
 BIC$_2$ &  &  &  &  &  &  &  1 &  99 &  &  &  &  &  &  &  & \\ 
 BIC$_3$ &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
 $F$ &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
 MDL &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
 UIP &  &  &  &  &  &  &  &  100 &  &  &  &  &  &  &  & \\ 
         \\
        &\multicolumn{16}{l}{\scriptsize{\textit{Colored Noise}}} \\
CV-Gabriel &  &  &  &  &  &  &  4 &  34 &  19 &  24 &  8 &  6 &  5 &  &  & \\ 
 RCV-Gabriel &  &  &  &  &  &  &  &  1 &  5 &  13 &  13 &  22 &  20 &  10 &  8 &  8\\ 
 CV-Wold &  &  &  &  &  1 &  1 &  8 &  83 &  4 &  3 &  &  &  &  &  & \\ 
 RCV-Wold &  &  &  &  &  &  &  &  26 &  32 &  25 &  12 &  2 &  2 &  &  &  1\\ 
 AIC &  &  &  &  &  &  &  &  &  &  &  &  &  2 &  10 &  19 &  69\\ 
 BIC$_1$ &  &  &  &  &  &  &  &  17 &  23 &  29 &  20 &  3 &  4 &  2 &  &  2\\ 
 BIC$_2$ &  &  &  &  &  &  &  &  24 &  36 &  27 &  9 &  2 &  1 &  1 &  & \\ 
 BIC$_3$ &  &  &  &  &  &  &  &  &  &  3 &  2 &  2 &  4 &  7 &  2 &  80\\ 
 $F$ &  &  &  &  &  &  &  &  &  4 &  9 &  32 &  28 &  14 &  11 &  &  2\\ 
 MDL &  &  &  &  &  &  &  &  16 &  24 &  31 &  19 &  6 &  2 &  1 &  &  1\\ 
 UIP &  &  &  &  &  &  &  &  &  &  4 &  17 &  27 &  23 &  18 &  9 &  2\\ 
        \\
        &\multicolumn{16}{l}{\scriptsize{\textit{Heavy Noise}}} \\
CV-Gabriel &  &  &  &  &  &  &  5 &  52 &  29 &  11 &  2 &  1 &  &  &  & \\ 
 RCV-Gabriel &  &  &  &  &  &  &  &  39 &  24 &  19 &  11 &  6 &  1 &  &  & \\ 
 CV-Wold &  &  &  &  &  1 &  1 &  11 &  83 &  4 &  &  &  &  &  &  & \\ 
 RCV-Wold &  &  &  &  &  &  &  &  66 &  28 &  5 &  1 &  &  &  &  & \\ 
 AIC &  &  &  &  &  &  &  &  2 &  21 &  24 &  33 &  11 &  6 &  3 &  & \\ 
 BIC$_1$ &  &  &  &  &  &  &  1 &  63 &  27 &  6 &  3 &  &  &  &  & \\ 
 BIC$_2$ &  &  &  &  &  &  &  1 &  71 &  22 &  5 &  1 &  &  &  &  & \\ 
 BIC$_3$ &  &  &  &  &  &  &  &  28 &  31 &  19 &  14 &  7 &  1 &  &  & \\ 
 $F$ &  &  &  &  &  &  &  &  32 &  41 &  14 &  12 &  1 &  &  &  & \\ 
 MDL &  &  &  &  &  &  &  &  63 &  28 &  6 &  3 &  &  &  &  & \\ 
 UIP &  &  &  &  &  &  &  &  20 &  32 &  27 &  15 &  6 &  &  &  & \\ 
        \bottomrule
    \end{tabular}
    \caption{
        \captiontitle{Rank estimation with strong sparse factors}
        Difference between the estimated rank and the true minimizer of 
        $\PE(k)$ for 100 replicates of strong sparse factors with various
        types of noise.
    }\label{T:rank-est-strong-sparse}
\end{table}

\begin{table}
    \tiny
    \centering    
    \begin{tabular}{lrrrrrrrrrrrrrrrr}
        \toprule
        &\multicolumn{15}{c}{\scriptsize{Estimated Rank}} \\
        \cmidrule{2-17}
        \scriptsize{Method}
            & $-7$ & $-6$ & $-5$ & $-4$ & $-3$ & $-2$ & $-1$ 
            & $\phantom{+}0$ 
            & $+1$ & $+2$ & $+3$ & $+4$ & $+5$ & $+6$ & $+7$ & $> 7$ \\
        \midrule
        \\
        &\multicolumn{16}{l}{\scriptsize{\textit{White Noise}}} \\
 CV-Gabriel &  &  &  &  &  &  &  3 &  56 &  32 &  9 &  &  &  &  &  & \\ 
 RCV-Gabriel &  &  &  &  &  &  &  5 &  66 &  20 &  8 &  1 &  &  &  &  & \\ 
 CV-Wold &  &  &  &  &  &  4 &  31 &  65 &  &  &  &  &  &  &  & \\ 
 RCV-Wold &  &  &  &  &  &  3 &  32 &  65 &  &  &  &  &  &  &  & \\ 
 AIC &  &  &  &  &  &  &  3 &  95 &  2 &  &  &  &  &  &  & \\ 
 BIC$_1$ &  &  &  &  &  1 &  13 &  46 &  40 &  &  &  &  &  &  &  & \\ 
 BIC$_2$ &  &  &  &  1 &  8 &  29 &  47 &  15 &  &  &  &  &  &  &  & \\ 
 BIC$_3$ &  &  &  &  &  &  &  &  99 &  1 &  &  &  &  &  &  & \\ 
 $F$ &  &  &  &  &  &  &  &  99 &  1 &  &  &  &  &  &  & \\ 
 MDL &  &  &  &  &  &  6 &  44 &  50 &  &  &  &  &  &  &  & \\ 
 UIP &  &  &  &  &  &  &  &  99 &  1 &  &  &  &  &  &  & \\ 
         \\
        &\multicolumn{16}{l}{\scriptsize{\textit{Colored Noise}}} \\
CV-Gabriel &  &  &  &  &  &  &  2 &  9 &  22 &  21 &  25 &  10 &  3 &  5 &  2 &  1\\ 
 RCV-Gabriel &  &  &  &  &  &  &  &  2 &  4 &  11 &  16 &  14 &  12 &  19 &  9 &  13\\ 
 CV-Wold &  1 &  4 &  4 &  7 &  9 &  17 &  14 &  34 &  4 &  &  &  &  2 &  &  1 &  1\\ 
 RCV-Wold &  &  &  &  &  &  &  2 &  29 &  24 &  17 &  8 &  6 &  7 &  2 &  4 &  1\\ 
 AIC &  &  &  &  &  &  &  &  &  &  &  &  &  4 &  14 &  14 &  68\\ 
 BIC$_1$ &  &  &  &  &  &  &  2 &  26 &  22 &  20 &  9 &  8 &  3 &  2 &  4 &  4\\ 
 BIC$_2$ &  &  &  &  &  &  &  4 &  39 &  28 &  12 &  5 &  4 &  1 &  3 &  3 &  1\\ 
 BIC$_3$ &  &  &  &  &  &  &  &  &  1 &  1 &  5 &  11 &  1 &  3 &  4 &  74\\ 
 $F$ &  &  &  &  &  &  &  &  1 &  7 &  16 &  24 &  17 &  13 &  10 &  1 &  11\\ 
 MDL &  &  &  &  &  &  &  1 &  25 &  23 &  21 &  8 &  9 &  4 &  1 &  4 &  4\\ 
 UIP &  &  &  &  &  &  &  &  &  2 &  11 &  12 &  22 &  19 &  15 &  4 &  15\\         
        \\
        &\multicolumn{16}{l}{\scriptsize{\textit{Heavy Noise}}} \\
CV-Gabriel &  &  &  &  1 &  &  1 &  2 &  10 &  51 &  25 &  8 &  1 &  1 &  &  & \\ 
 RCV-Gabriel &  &  &  &  &  &  &  &  13 &  37 &  24 &  13 &  9 &  2 &  1 &  &  1\\ 
 CV-Wold &  1 &  4 &  6 &  4 &  13 &  17 &  21 &  32 &  1 &  &  &  &  &  &  1 & \\ 
 RCV-Wold &  &  &  &  &  1 &  2 &  13 &  59 &  16 &  4 &  2 &  1 &  1 &  &  &  1\\ 
 AIC &  &  &  &  &  &  &  &  7 &  15 &  28 &  25 &  14 &  7 &  1 &  2 &  1\\ 
 BIC$_1$ &  &  &  &  &  1 &  2 &  16 &  58 &  16 &  3 &  2 &  &  1 &  &  &  1\\ 
 BIC$_2$ &  &  &  &  &  4 &  15 &  23 &  46 &  8 &  1 &  1 &  1 &  &  1 &  & \\ 
 BIC$_3$ &  &  &  &  &  &  &  &  38 &  21 &  23 &  8 &  4 &  4 &  1 &  &  1\\ 
 $F$ &  &  &  &  &  &  &  &  45 &  22 &  19 &  11 &  1 &  1 &  &  &  1\\ 
 MDL &  &  &  &  &  &  2 &  14 &  61 &  15 &  3 &  3 &  &  1 &  &  &  1\\ 
 UIP &  &  &  &  &  &  &  &  27 &  29 &  27 &  9 &  4 &  3 &  &  &  1\\ 
        \bottomrule
    \end{tabular}
    \caption{
        \captiontitle{Rank estimation with weak Gaussian factors}
        Difference between the estimated rank and the true minimizer of 
        $\PE(k)$ for 100 replicates of weak Gaussian factors with various
        types of noise.
    }\label{T:rank-est-weak-gauss}
\end{table}


\begin{table}
    \tiny
    \centering
    \begin{tabular}{lrrrrrrrrrrrrrrrr}
        \toprule
        &\multicolumn{15}{c}{\scriptsize{Estimated Rank}} \\
        \cmidrule{2-17}
        \scriptsize{Method}
            & $-7$ & $-6$ & $-5$ & $-4$ & $-3$ & $-2$ & $-1$ 
            & $\phantom{+}0$ 
            & $+1$ & $+2$ & $+3$ & $+4$ & $+5$ & $+6$ & $+7$ & $> 7$ \\
        \midrule
        \\
        &\multicolumn{16}{l}{\scriptsize{\textit{White Noise}}} \\
CV-Gabriel &  &  &  &  &  &  8 &  16 &  39 &  27 &  8 &  2 &  &  &  &  & \\ 
 RCV-Gabriel &  &  &  &  &  &  1 &  8 &  53 &  27 &  9 &  1 &  1 &  &  &  & \\ 
 CV-Wold &  &  &  2 &  2 &  11 &  16 &  29 &  40 &  &  &  &  &  &  &  & \\ 
 RCV-Wold &  &  &  &  &  2 &  15 &  29 &  54 &  &  &  &  &  &  &  & \\ 
 AIC &  &  &  &  &  &  &  6 &  77 &  16 &  1 &  &  &  &  &  & \\ 
 BIC$_1$ &  &  &  &  1 &  4 &  21 &  45 &  29 &  &  &  &  &  &  &  & \\ 
 BIC$_2$ &  &  &  1 &  4 &  14 &  27 &  37 &  17 &  &  &  &  &  &  &  & \\ 
 BIC$_3$ &  &  &  &  &  &  1 &  2 &  86 &  11 &  &  &  &  &  &  & \\ 
 $F$ &  &  &  &  &  &  &  &  85 &  15 &  &  &  &  &  &  & \\ 
 MDL &  &  &  &  &  2 &  19 &  42 &  37 &  &  &  &  &  &  &  & \\ 
 UIP &  &  &  &  &  &  &  &  73 &  26 &  1 &  &  &  &  &  & \\ 
         \\
        &\multicolumn{16}{l}{\scriptsize{\textit{Colored Noise}}} \\
CV-Gabriel &  &  &  &  &  &  2 &  5 &  16 &  19 &  19 &  12 &  9 &  3 &  10 &  3 &  2\\ 
 RCV-Gabriel &  &  &  &  &  &  &  &  1 &  2 &  7 &  12 &  15 &  19 &  11 &  8 &  25\\ 
 CV-Wold &  1 &  3 &  6 &  11 &  10 &  17 &  21 &  25 &  2 &  2 &  &  &  1 &  &  &  1\\ 
 RCV-Wold &  &  &  &  &  &  &  2 &  22 &  21 &  19 &  12 &  11 &  3 &  5 &  3 &  2\\ 
 AIC &  &  &  &  &  &  &  &  &  &  &  &  &  2 &  8 &  9 &  81\\ 
 BIC$_1$ &  &  &  &  &  &  &  1 &  22 &  23 &  20 &  10 &  9 &  4 &  2 &  3 &  6\\ 
 BIC$_2$ &  &  &  &  &  1 &  &  7 &  37 &  17 &  13 &  9 &  7 &  1 &  3 &  4 &  1\\ 
 BIC$_3$ &  &  &  &  &  &  &  &  &  &  2 &  2 &  2 &  4 &  8 &  7 &  75\\ 
 $F$ &  &  &  &  &  &  &  &  &  4 &  6 &  21 &  22 &  14 &  13 &  6 &  14\\ 
 MDL &  &  &  &  &  &  &  &  22 &  19 &  22 &  12 &  10 &  5 &  2 &  3 &  5\\ 
 UIP &  &  &  &  &  &  &  &  &  &  5 &  9 &  22 &  19 &  17 &  10 &  18\\ 
        \\
        &\multicolumn{16}{l}{\scriptsize{\textit{Heavy Noise}}} \\
CV-Gabriel &  &  &  &  &  &  2 &  4 &  22 &  25 &  24 &  20 &  3 &  &  &  & \\ 
 RCV-Gabriel &  &  &  &  &  &  1 &  4 &  9 &  18 &  20 &  18 &  12 &  13 &  2 &  1 &  2\\ 
 CV-Wold &  1 &  1 &  4 &  10 &  16 &  19 &  29 &  18 &  &  1 &  &  &  &  &  1 & \\ 
 RCV-Wold &  &  &  &  &  &  2 &  19 &  51 &  7 &  16 &  3 &  1 &  &  &  1 & \\ 
 AIC &  &  &  &  &  &  &  &  2 &  17 &  16 &  19 &  22 &  8 &  4 &  10 &  2\\ 
 BIC$_1$ &  &  &  &  &  1 &  4 &  21 &  51 &  5 &  13 &  3 &  1 &  &  &  1 & \\ 
 BIC$_2$ &  &  &  &  2 &  4 &  16 &  32 &  31 &  4 &  9 &  &  1 &  &  &  1 & \\ 
 BIC$_3$ &  &  &  &  &  &  &  3 &  25 &  23 &  17 &  12 &  10 &  4 &  4 &  1 &  1\\ 
 $F$ &  &  &  &  &  &  &  2 &  29 &  21 &  23 &  10 &  7 &  5 &  2 &  &  1\\ 
 MDL &  &  &  &  &  &  3 &  19 &  52 &  6 &  15 &  3 &  1 &  &  &  1 & \\ 
 UIP &  &  &  &  &  &  &  &  20 &  23 &  22 &  13 &  12 &  3 &  5 &  1 &  1\\ 
        \bottomrule
    \end{tabular}
    \caption{
        \captiontitle{Rank estimation with weak sparse factors}
        Difference between the estimated rank and the true minimizer of 
        $\PE(k)$ for 100 replicates of weak sparse factors with various
        types of noise.
    }\label{T:rank-est-weak-sparse}
\end{table}

\clearpage

\section{Real data example}

We conclude this chapter with a neuroscience application. The Neural
Prosthetic Systems Laboratory at Stanford University (NPSL) is interested in
studying the motor cortex region of the brain. Essentially, they want to know
what the correspondence is between neural activity in that part of the brain
and motor activity (movement). Research is still at a very fundamental level,
and the basic question of how many things are being represented is still
unanswered. It is thought that desired position, speed, and velocity get
expressed as neural activity, but conjectures about the dimensionality of the
neural responses vary from $7$ to $20$ or more.

NPSL has designed and carried out an experiment meant to measure the 
dimensionality of neural response for a two-dimensional motion task.  The 
experiment involves measuring the activity in $49$ neurons as a monkey 
performs $27$ different movement tasks (conditions).

For a particular neuron and condition, a simplified explanation of the 
experiment is as follows:
\begin{enumerate}
    \item At time $t=0 \, \text{ms}$, start recording neural activity in the 
        monkey.
    \item At time $t=400 \, \text{ms}$ (\textsc{target-on}), show the monkey a 
        target.  The monkey is not allowed to move at this point.
    \item At a random time between time $t=400 \, \text{ms}$ and time 
        $t = 1560 \, \text{ms}$, allow the monkey to move.
    \item At time $t=1560 \, \text{ms}$ (\textsc{movement}), the monkey starts 
        to move and point at the target.
    \item Record activity up to but not including time $t=2110 \, \text{ms}$. 
\end{enumerate}
Each condition includes a target position and a configuration of obstacles. 
The same monkey is used for every trial.  Measurements are taken at 
$5 \, \text{ms}$ intervals, so that there are $422$ total time points.  It is 
necessary to do some registration, scaling, and interpolation before doing 
more serious data analysis, but the details of those processes are not 
important for our purposes.  Figure~\ref{F:motor-cortex-data} shows the 
preprocessed responses for each neuron and condition.

\begin{figure}[tbh]
    \centering
    \begin{minipage}{0.38\textwidth}
        \begin{center}
            \includegraphics{neuron1}
        \end{center}
    \end{minipage}
    \begin{minipage}{0.61\textwidth}
        \begin{center}
            \includegraphics{neurons}
        \end{center}
    \end{minipage}
    \caption{
        \captiontitle{Motor cortex data}
        Response rates in 47 neurons for 27 movement tasks.  The subplots show
        the normalized response rates in a single neuron as functions of time.  
        Each color corresponds to a different movement task.  The plot on
        the left is a zoomed-in view of the data for the first neuron.
    }\label{F:motor-cortex-data}
\end{figure}

The data from the NPSL motor cortex experiment can be put into a matrix where
each neuron is thought of as a variable, and the timepoints of each condition
are thought of as observations. This gives us a matrix $\mX$ with $p = 49$
variables and $n = 27 \cdot 422 = 11394$ observations. Of course, the rows of
$\mX$ are nothing like iid, and the noise in $\mX$ is not white. Parametric
methods are not likely to give very reliable estimates of the dimensionality
of $\mX$, but cross-validation stands a reasonable chance.

After centering the columns of $\mX$, we performed Wold- and Gabriel-style cross-validation to estimate the dimensionality of the signal part of $\mX$.  
For Gabriel-style CV, we first tried both $(2,2)$-fold and $(2,49)$-fold; both resulting $\widehat{PE}(k)$ curves had their minima at the maximum $k$.  For $5$-fold Wold-style CV, there is a minimum at $k=13$.  The BIC, $F$, MDL, and UIP estimators all chose $k=48$ as the dimensionality, while the AIC estimator chose $k=47$.  We show the cross-validation estimated prediction curves in Figure~\ref{F:motor-cv-est}.  It is likely that the true dimensionality of $\mX$ is high.

\begin{figure}
    \centering
    \includegraphics{motor-cv-est}
    \caption{
        \captiontitle{Motor cortex estimated prediction error}
        Prediction error as a function of rank, estimated by three 
        cross-validation methods.  The units are normalized so that the
        maximum prediction error is $1.0$.  Error bars show one standard    
        error, estimated from the folds. The prediction error estimate from
        Wold-style CV shows a minimum at $k=13$, but the Gabriel-style
        CV estimates always decrease with $k$.  It is likely that the true 
        dimensionality of the data is high.
    }\label{F:motor-cv-est}
\end{figure}


\section{Summary and future work}

We have described two different forms of cross-validation appropriate for
model selection in unsupervised learning. Wold-style CV uses a ``speckled''
leave-out, and Gabriel-style CV uses a ``blocked'' leave-out. We have defined
two forms of error associated with SVD/PCA-like models, the prediction error
and the model error. Through simulations, we have shown that both forms of CV
can be considered to give estimates of prediction error. Both methods perform
well, but Wold-style CV seems to be more robust to badly-behaved noise. We
have applied these cross-validation methods to a data analysis problem from a
neuroscience experiment.

We have focused on latent factor models and the singular value decomposition.
However, it is relatively easy to translate the two styles of cross-validation
presented here to other unsupervised learning methods. For Wold-style
hold-outs, an EM-like algorithm can usually be applied to the models in many
unsupervised learning contexts. The Gabriel-style philosophy of ``treat some
of the variables as response and the others as predictors'' can also be
applied more broadly. We are in the process of investigating both
cross-validation strategies for clustering and kernel-based manifold learning.

This chapter leaves a number of open questions.  Mainly, we have not provided 
any theoretical results here, only simulations.  A quote from Downton's 
discussion of Stone's 1974 paper on cross-validation equally applies to our 
work:
\begin{quote}
    A current nine-day wonder in the press concerns the exploits of a Mr. Uri
    Geller who appears to be able to bend metal objects without touching them;
    [The author] seems to be attempting to bend statistics without touching
    them. My attitude to both of these phenomena is one of open-minded 
    scepticism; I do not believe in either of these prestigious activities, on 
    the other hand they both deserve serious scientific examination
    \cite{stone1974cv}.
\end{quote}
Despite Dowton's skepticism, cross-validation has proven to be an invaluable tool for supervised learning.  It is our hope that with some additional work, CV can be just as valuable for unsupervised learning.  In the next chapter, we provide some theoretical justification for Gabriel-style cross-validation, but analysis of Wold-style cross-validation is still an open problem.


intrinsic-rank.tex

\chapter{An intrinsic notion of rank for factor models}
\label{C:intrinsic-rank}

As Moore's Law progresses, data sets measuring on the order of hundreds or
thousands of variables are becoming increasingly more common. Making sense of
data of this size is simply not tractable without imposing a simplifying
model. One popular simplification is to posit existence of a small number of
common factors that drive the dynamics of the data, which are usually
estimated by principal component analysis (PCA) or some variation thereof.
The $n \times p$ data matrix $\mX$ is approximated as a low-rank product $\mX
\approx \mU \mV^\trans$, where $\mU$ and $\mV$ are $n \times k$ and $p \times
k$, respectively, with $k$ much smaller than $n$ and $p$.

The number of algorithms for approximating matrices by low-rank products has
exploded in recent years. These algorithms include archetypal analysis
\cite{cutler1994aa}, the semi-discrete decomposition (SDD)
\cite{kolda1998smd}, the non-negative matrix factorization (NMF)
\cite{lee1999lpo}, the plaid model \cite{lazzeroni2002pmg}, the $CUR$
decomposition \cite{drineas2007fmc}, and regularized versions thereof. They
also include some clustering methods, in particular $k$-means and fuzzy
$k$-means \cite{bezdek1980fmp}.

A prevailing question is: How many common factors underlie a data set? Alternately, how should one choose $k$? In general, the answer to this question is
application-specific. If we are trying to use $\mX$ to predict a response,
$y$, then the optimal $k$ is the one that gives the best prediction error
for $y$. The situation is not always this simple, though. For exploratory
analysis, there is no external response, and we want to choose a $k$ that is
``intrinsic'' to $\mX$. For other applications, we don't have a single
response, $y$, we have \emph{many} responses $y_1, y_2, \ldots, y_m$.
We may not even know all of the $y_i$ when we are processing $\mX$. We want a $k$ that has good average-case or worst-case prediction properties for a large class of responses.

In this chapter, we develop a precise notion of intrinsic rank for ``latent
factor'' matrix data. This choice of $k$ is the one that minimizes average- or
worst-case prediction error over all bilinear statistics. We specifically work
with the singular value decomposition (SVD), but our definition can be
extended to other matrix factorizations as well.

Section~\ref{S:latent-factor-model} introduces the latent factor model, the
data model from which we base our constructions.
Section~\ref{S:intrinsic-rank} defines intrinsic rank as the minimizer of a
loss function. We give a theoretical analysis of the behavior of some natural
loss functions in Section~\ref{S:loss-behavior}, followed by simulations in
Section~\ref{S:intrinsic-loss-simulations}. In
Section~\ref{S:intrinsic-rank-scree-plot}, we examine the connection between
intrinsic rank and the scree plot. Finally,
Section~\ref{S:intrinsic-rank-extensions} discusses some extensions and
Section~\ref{S:intrinsic-rank-summary} gives a summary of the chapter.


\section{The latent factor model}\label{S:latent-factor-model}

We start by describing a model for data generated by a small number of latent
factors and additive noise. Suppose that we have $n$ multivariate observations
$\vx_{1}, \vx_{2}, \ldots, \vx_{n} \in \reals^p$. In a microarray setting, we
will have about $n=50$ arrays measuring the activations of around $p=5000$ genes (or $50000$, or even millions of genes in the case of exon arrays). Alternatively, for financial applications $\vx_i$ will measure the
market value of on the order of $p=1000$ assets on day $i$, and we may be looking at
data from the last three years, so $n \approx 1000$. In these situations and
others like them, we can often convince ourselves that there aren't really
$5000$ or $1000$ different things going on in the data. Probably, there are a small number, $k$ of unobserved factors driving the dynamics of the data. Typically, in fact, we think $k$ is on the order of around $5$ or $10$.

To be specific about this intuition, for genomics applications, we don't
really think that all $p=5000$ measured genes are behaving independently. On
the contrary, we think that there are a small number of biological processes
that determine how much of each protein gets produced. In finance, while it is
true that the stock prices of individual companies have a certain degree of
independence, often macroscopic effects like industry- and market-wide trends
can explain a substantial portion of the value.

\subsection{The spiked model}

One way to model latent effects is to assume that the $\vx_i$ are iid and
that their covariance is ``spiked''. We think that $\vx_i$ is a weighted
combination of $k$ latent factors corrupted by additive white noise. In this
case, $\vx_i$ can be decomposed as
\begin{align}
    \vx_i
        &=
        \sum_{j=1}^k
            \va_j
            s_{i,j}
        +
        \vepsilon_i \notag \\
        &=
        \mA \vs_i + \vepsilon_i,
\end{align}
where
\(
    \mA
    =
    \left(
    \begin{matrix}
        \va_1 & \va_2 & \cdots & \va_k
    \end{matrix}
    \right)
\)
is a $p \times k$ matrix of latent factors common to all observations and
$\vs_i \in \reals^{k}$ is a vector of loadings for the $i$th observation. We assume that the
noise vector $\vepsilon_i$ is distributed as $\Normal( 0, \, \sigma^2 \mI_p
)$. If the loadings have mean zero and covariance $\mSigma_\text{S} \in
\reals^{k \times k}$, and if they are also independent of the noise, then
$\vx_i$ has covariance
\begin{equation}\label{E:spiked-cov-natural-form}
    \mSigma
        \define
        \E \left[ \vx_i \, \vx_i^\trans \right]
            =
                \mA \mSigma_\text{S} \mA^\trans
                +
                \sigma^2
                \mI_p.
\end{equation}
The decomposition in \eqref{E:spiked-cov-natural-form} can be reparametrized
as
\begin{equation}\label{E:spiked-cov-identifiable-form}
    \mSigma
        =
        \mQ \mLambda \mQ^\trans
        +
        \sigma^2
        \mI_p,
\end{equation}
where $\mQ^\trans \mQ = \mI_k$ and
\(
    \mLambda
        = 
        \diag \left( \lambda_1, \lambda_2, \ldots, \lambda_k \right),
\)
with $\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_k \geq 0$.
Equation~\eqref{E:spiked-cov-identifiable-form} makes it apparent that
$\mSigma$ is ``spiked'', in the sense that most of its eigenvalues are equal,
but $k$ eigenvalues stand out above the bulk. The first $k$ eigenvalues are
$\lambda_1 + \sigma^2, \lambda_2 + \sigma^2, \ldots, \lambda_k + \sigma^2$,
and the remaining $p-k$ eigenvalues are equal to $\sigma^2$.

\subsection{More general matrix models}

We can introduce a model more general than the spiked one by specifying
a distribution for the $n\times p$ data matrix 
\(
    \mX
    =
    \left(
    \begin{matrix}
        \vx_1 &
        \vx_2 &
        \cdots &
        \vx_n
    \end{matrix}
    \right)^\trans
\)
that includes dependence between the rows.  In the spiked model, the
distribution of $\mX$ can be described as
\begin{equation}\label{E:data-matrix-spiked}
    \mX
        \eqd
        \mZ
        \mLambda^{1/2}
        \mQ^\trans
        +
        \mE,
\end{equation}
where 
\(
    \mE 
        = 
        \left(
        \begin{matrix}
            \vepsilon_1 & \vepsilon_2 & \cdots & \vepsilon_n
        \end{matrix}
        \right)^\trans
\)
and $\mZ$ is an $n \times k$ matrix of independent 
$\Normal \left( 0, \, 1 \right)$ elements.  More generally, we can consider
data of the form
\begin{equation}\label{E:data-matrix-general}
    \mX
        \eqd
            \sqrt{n} \,
            \mU
            \mD
            \mV^\trans
            +
            \mE,
\end{equation}
where $\mU^\trans \mU = \mV^\trans \mV = \mI_k$, and $\mD = \diag( d_1, d_2,
\ldots, d_k )$ with $d_1 \geq d_2 \geq \cdots \geq d_k \geq 0$. We can get
\eqref{E:data-matrix-general} from \eqref{E:data-matrix-spiked} by letting
$\mZ \mLambda^{1/2} = \sqrt{n} \, \mU \mD \mC^\trans$ be the SVD of $\mZ
\mLambda^{1/2}$ and defining $\mV = \mQ \mC$. Unlike the spiked model,
\eqref{E:data-matrix-general} can model dependence between variables as well
as dependence between observations.

\section{An intrinsic notion of rank}\label{S:intrinsic-rank}

With Section~\ref{S:latent-factor-model}'s latent factor model in mind, we
turn our attention to defining the intrinsic rank of a data set. This
definition will be motivated both by the generative model for $\mX$ and by
predictive power considerations.  When 
\(
    \mX = \sqrt{n} \, \mU \mD \mV^\trans + \mE,
\)
we think of $\sqrt{n} \, \mU \mD \mV^\trans$ as ``signal'' and $\mE$ as ``noise''.  We make a distinction between the generative rank and the effective rank.

\begin{definition}
    If the $n \times p$ matrix $\mX$ is distributed as
    \(
        \mX = \sqrt{n} \, \mU \mD \mV^\trans + \mE,
    \)
    where $\mU^\trans \mU = \mV^\trans \mV = \mI_{k_0}$, $\mD$ is a
    $k_0 \times k_0$ diagonal matrix with positive diagonal entries, and $\mE$
    is a noise matrix independent of the signal term whose elements are
    iid $\Normal( 0, \, \sigma^2 )$, then
    we denote by $k_0$ the \emph{generative rank} of $\mX$.
\end{definition}

\noindent 
Intuitively the generative rank is the rank of the signal part of $\mX$.

The effective rank is defined in terms of how well the first terms of the
SVD of $\mX$ approximates the signal $\sqrt{n} \, \mU \mD \mV^\trans$.  We
let $\mX = \sqrt{n} \mhU \mhD \mhV^\trans$ be the full SVD of $\mX$, where
\(
    \mhU
    =
    \left(
    \begin{matrix}
        \vhu_1 & \vhu_2 & \cdots & \vhu_{n\wedge p}
    \end{matrix}
    \right),
\)
\(
    \mhV
    =
    \left(
    \begin{matrix}
        \vhv_1 & \vhv_2 & \cdots & \vhv_{n\wedge p}
    \end{matrix}
    \right),
\)
and
\(
    \mhD
    =
    \diag \left(
        \hd_1, \hd_2, \ldots, \hd_{n\wedge p}
    \right).
\)
If we let
\(
    \mhD(k)
    =
    \diag \left(
        \hd_1, \hd_2, \ldots, \hd_k, 0, 0, \ldots, 0
    \right),
\)
then the SVD truncated to $k$ terms is
\(
    \mhX(k)
    =
    \sqrt{n} \,
    \mhU \mhD(k) \mhV^\trans.
\)
We are now in a position to define effective rank

\begin{definition}
    Given a loss function
    \(
        L : \reals^{n \times p} \times \reals^{n \times p} \to \reals,
    \)
    the effective rank of $\mX$ with respect to $L$ is equal to
    \begin{equation}
        k_L^\ast
        \define
        \argmin_k \, 
            L \left( 
                \sqrt{n} \mU \mD \mV^\trans\!, \, \mhX(k) 
            \right).
    \end{equation}
\end{definition}

The effective rank depends on the choice of loss function.  In some settings it is preferable to choose an application-specific loss function.  Often, we appeal to simplicity and convenience and choose squared Frobenius loss.  Specifically,
\begin{equation}
    L_\Frob ( \mA, \, \mA' )
        = \| \mA - \mA' \|_\Frob^2.
\end{equation}
One way to motivate this loss function is that it measures average squared error over all bilinear statistics of the form 
$\valpha^\trans \mA \, \vbeta$,
\[
    \frac{1}{np} \,
    \| \mA - \mA' \|_\Frob^2
    =
    \int\limits_{ \substack{ \| \valpha \|_2 = 1, \\
                             \| \vbeta \|_2  = 1 } }
        \!\!\!\!
        \left(
            \valpha^\trans \mA \, \vbeta
            -
            \valpha^\trans \mA' \vbeta
        \right)^2
        d\valpha \,
        d\vbeta
\]
(see Section~\ref{S:probabilistic-frob} for details).
In the context of the latent factor model, the effective rank with respect
to $L_\Frob$ is the rank that gives the best average-case predictions of 
bilinear statistics of the signal part (with respect to squared-error loss). 

A common alternative to Frobenius loss is spectral loss, given by
\begin{equation}
    L_2 ( \mA, \, \mA' )
        = \| \mA - \mA' \|_2^2.
\end{equation}
This can be interpreted as worst-case squared error over the class of all 
bilinear statistics,
\[
    \| \mA - \mA' \|_2^2
        =
            \sup_{ \substack{ \| \valpha \|_2 = 1, \\
                              \| \vbeta \|_2  = 1 } }
                \left(
                    \valpha^\trans \mA \, \vbeta
                    -
                    \valpha^\trans \mA' \vbeta
                \right)^2.
\]
In the sequel, we denote the optimal ranks with respect to Frobenius and
spectral loss as $k^\ast_\Frob$ and $k^\ast_2$, respectively.


\section{Loss behavior}\label{S:loss-behavior}

In this section we investigate the behavior of the loss functions introduced
in Section~\ref{S:intrinsic-rank}.  First, we need to be more precise about
our working assumptions on the data matrices.  The theory is easier if we
work in an asymptotic setting, introducing a sequence of data matrices
$\mX_1, \mX_2, \ldots, \mX_n$, where $\mX_n \in \reals^{n \times p}$, $p = p(n)$, $n \to \infty$, and $\frac{n}{p} \to \gamma \in (0,\infty)$.  We
will need three assumptions.

\begin{assumption}
    The matrix $\mX_n \in \reals^{n \times p}$ can be decomposed as
    \begin{equation}
        \mX_n = \sqrt{n} \, \mU_n \mD_n \mV_n^\trans + \mE_n.
    \end{equation}
    Here, 
    $\mU_n \in \reals^{n \times k_0}$, $\mD_n \in \reals^{k_0 \times k_0}$,
    and $\mV_n \in \reals^{p \times k_0}$.  The left and right factors
    $\mU_n$ and $\mV_n$ satisfy
    \(
        \mU_n^\trans \mU_n = \mV_n^\trans \mV_n = \mI_{k_0}.
    \)
    The aspect ratio satisfies
    $\frac{n}{p} = \gamma + \oh\left( \frac{1}{\sqrt{n}}\right)$ for a
    constant $\gamma \in (0, \infty)$.  The number of factors $k_0$
    is fixed.
\end{assumption}

\begin{assumption}
    The matrix of normalized factor strengths is diagonal with
    \(
        \mD_n
            =
                \diag \left(
                    d_{n,1}, d_{n,2}, \ldots, d_{n,k_0}
                \right).
    \)
    For $1 \leq i \leq k_0$, the diagonal elements satisfy
    $d_{n,i}^2 \toas \mu_i$ for deterministic $\mu_i$ satisfiying
    \(
        \mu_1 > \mu_2 > \cdots > \mu_{k_0} > 0.
    \)
\end{assumption}

\begin{assumption}
    The noise matrix $\mE_n$ has iid elements independent of $\mU_n$, 
    $\mD_n$, and $\mV_n$, with $E_{n,11} \sim \Normal( 0, \, \sigma^2)$.
\end{assumption}

\noindent
These assumptions allow us to apply the results of 
Chapter~\ref{C:svd-behavior} to get the first-order behavior of the
SVD of $\mX_n$.  As before, we let $\vu_{n,1}, \vu_{n,2}, \ldots, \vu_{n,k_0}$
and $\vv_{n,1}, \vv_{n,2}, \ldots, \vv_{n,k_0}$ denote the columns of
$\mU_n$ and $\mV_n$, respectively.  We set 
$\mX_n = \sqrt{n} \, \mhU_n \mhD_n \mhV_n^\trans$ to be the SVD of $\mX_n$,
where the columns of $\mhU_n$ and $\mhV_n$ are
\(
    \vhu_{n,1}, \vhu_{n,2}, \ldots, \vhu_{n,n\wedge p}
\)
and
\(
    \vhv_{n,1}, \vhv_{n,2}, \ldots, \vhv_{n,n\wedge p},
\)
respectively.  With
\(
    \mhD_n 
        = 
            \diag \left(
                \hmu_{n,1}^{1/2}, 
                \hmu_{n,2}^{1/2}, 
                \ldots, 
                \hmu_{n,n\wedge p}^{1/2}
            \right),
\)
we set
\(
    \mhD_n(k)
        =
            \diag \left(
                \hmu_{n,1}^{1/2}, 
                \hmu_{n,2}^{1/2}, 
                \ldots, 
                \hmu_{n,k}^{1/2}, 
                0, 
                0,
                \ldots, 
                0
            \right)
\)
so that
\(
    \mhX_{n}(k) = \sqrt{n} \, \mhU_n \mhD_n(k) \mhV_n^\trans
\)
is the SVD of $\mX_n$ truncated to $k$ terms.

We can decompose the columns of $\mhV_n$ into sums of two terms, with the first term in the subspace spanned by $\mV_n$, and the second term orthogonal
to it.  By setting $\mTheta_n = \mV_n^\trans \mhV_n$ and 
taking the $QR$ decomposition of $\mhV_n - \mV_n \mTheta_n$, the matrix of right factors can be expanded as
\begin{equation}\label{E:v-vbar-decomp}
    \mhV_n = \mV_n \mTheta_n + \mbV_n \mbTheta_n,
\end{equation}
with $\mbV_n \in \reals^{p \times (p-k_0)}$  satisfying
$\mbV_n^\trans \mbV_n = \mI_{p-k_0}$ and $\mV_n^\trans \mbV_n = 0$.  We choose the signs so that $\mbTheta_n$ has non-negative diagonal entries.  Note that
\begin{equation}
    \mI_k
    =
        \mhV_n^\trans \mhV_n
        =
        \mTheta_n^\trans \mTheta_n
        +
        \mbTheta_n^\trans
        \mbTheta_n.
\end{equation}
In particular, since $\mbTheta_n$ is upper-triangular, if $\mTheta_n$ converges to a diagonal matrix $\mTheta$, then $\mbTheta_n$ must also, converge to a diagonal matrix,
\(
    \left(
        \mI_k
        -
        \mTheta^2
    \right)^{1/2}.
\)
This makes the decomposition in equation~\eqref{E:v-vbar-decomp} very
convenient to work with.

The same trick applies to $\mhU_n$.  We expand
\begin{equation}\label{E:u-ubar-decomp}
    \mhU_n = \mU_n \mPhi_n + \mbU_n \mbPhi_n,
\end{equation}
with $\mbU_n$ and $\mbPhi_n$ defined analogously to $\mbV_n$ and $\mbTheta_n$.
Again, we have that
\begin{equation}
    \mI_k
        =
        \mPhi_n^\trans \mPhi_n
        +
        \mbPhi_n^\trans
        \mbPhi_n.    
\end{equation}
Likewise, if $\mPhi_n$ converges to a diagonal matrix, then $\mbPhi_n$ must
do the same.

We can new get a simplified formula for $\mhX_n(k)$.  With the decompositions
in equations~\eqref{E:v-vbar-decomp}~and~\eqref{E:u-ubar-decomp}, we get
\begin{equation}
    \mhX_n(k)
        =
        \sqrt{n} \,
        \left(
        \begin{matrix}
            \mU_n & \mbU_n
        \end{matrix}
        \right)
        \left(
        \begin{matrix}
            \mPhi_n \mhD_n(k) \mTheta_n^\trans &
                \mPhi_n \mhD_n(k) \mbTheta_n^\trans \\
            \mbPhi_n \mhD_n(k) \mTheta_n^\trans &
                \mbPhi_n \mhD_n(k) \mbTheta_n^\trans
        \end{matrix}
        \right)
        \left(
        \begin{matrix}
            \mV_n^\trans \\
            \mbV_n^\trans
        \end{matrix}
        \right).
\end{equation}

We can get the asymptotic limits of the quantities above.
For $1 \leq i \leq k_0$, we set
\begin{subequations}
\begin{align}
    \bmu_i
        &=
        \begin{cases}
            \left( \mu_i + \sigma^2 \right)
            \left( 1 + \frac{\sigma^2}{\gamma \mu_i} \right)
                &\text{when $\mu_i > \frac{\sigma^2}{\sqrt{\gamma}}$}, \\
            \sigma^2 \left( 1 + \frac{1}{\sqrt{\gamma}} \right)^2
                &\text{otherwise,}
        \end{cases} \\
    \theta_i 
        &=
        \begin{cases}
            \sqrt{ 
                \left( 1 - \frac{\sigma^4}{ \gamma \mu_i^2} \right) 
                \left( 1 + \frac{\sigma^2}{ \gamma \mu_i  } \right)^{-1} }
            &\text{when $\mu_i > \frac{\sigma^2}{\sqrt{\gamma}}$,} \\
            0
            &\text{otherwise,}
        \end{cases} \\
    \varphi_i
        &=
        \begin{cases}
            \sqrt{
                \left( 1 - \frac{\sigma^4}{ \gamma \mu_i^2} \right)
                \left( 1 + \frac{\sigma^2}{ \mu_i  } \right)^{-1} }
            &\text{when $\mu_i > \frac{\sigma^2}{\sqrt{\gamma}}$,} \\
            0
            &\text{otherwise,}
        \end{cases}
\end{align}
while for $i > k_0$, we set 
$\bmu_i = \sigma^2 \left( 1 + \frac{1}{\sqrt{\gamma}} \right)^2$ 
and $\theta_i = \varphi_i = 0$.  For $i \geq 1$, we
define
\begin{align}
    \bar \theta_i  &= \sqrt{ 1 - \theta_i^2 }, \\
    \bar \varphi_i &= \sqrt{ 1 - \varphi_i^2 }.
\end{align}
\end{subequations}
With
\begin{subequations}
\begin{equation}
    \mD(k) 
        = 
            \diag\left( 
                \bmu_1^{1/2}, 
                \bmu_2^{1/2}, 
                \ldots,
                \bmu_k^{1/2},
                0,
                0,
                \ldots,
                0
            \right) \in \reals^{n \times p}
\end{equation}
and
\begin{align}
    \mTheta
        &=
            \diag\left(
                \theta_1,
                \theta_2,
                \ldots,
                \theta_p
            \right), \\
    \mPhi
        &=
            \diag\left(
                \varphi_1,
                \varphi_2,
                \ldots,
                \varphi_n
            \right), \\
    \mbTheta
        &=
            \diag\left(
                \bar \theta_1,
                \bar \theta_2,
                \ldots,
                \bar \theta_p
            \right), \\
    \mbPhi
        &=
            \diag\left(
                \bar \varphi_1,
                \bar \varphi_2,
                \ldots,
                \bar \varphi_n
            \right),
\end{align}
\end{subequations}
Theorems~\ref{T:spiked-eigenvalue-limits}~and~\ref{T:spiked-eigenvector-limits}
give us that for fixed $k$ as $n \to \infty$,
\begin{align*}
    \mPhi_n \mhD_n(k) \mTheta_n^\trans
        &\toas
            \mPhi \mD(k) \mTheta^\trans, \\
    \mPhi_n \mhD_n(k) \mbTheta_n^\trans
        &\toas
            \mPhi \mD(k) \mbTheta^\trans, \\
    \mbPhi_n \mhD_n(k) \mTheta_n^\trans
        &\toas
            \mbPhi \mD(k) \mTheta^\trans, \\
    \mbPhi_n \mhD_n(k) \mbTheta_n^\trans
        &\toas
            \mbPhi \mD(k) \mbTheta^\trans.
\end{align*}
This result makes it easy to analyze the loss behavior.  Letting $\mu_i = 0$
for $i > k_0$, putting $\bmu_i(k) = \bmu_i \, 1\{i \leq k\}$ and
\begin{equation}
    \mF_i(k)
        =
            \left(
            \begin{matrix}
                \mu_i^{1/2} - \varphi_i \, \bmu_i^{1/2}(k) \, \theta_i &
                    -\varphi_i \, \bmu_i^{1/2}(k) \, \bar \theta_i \\
                -\bar \varphi_i \, \bmu_i^{1/2}(k) \, \theta_i &
                    -\bar \varphi_i \, \bmu_i^{1/2}(k) \, \bar \theta_i
            \end{matrix}
            \right)
\end{equation}
we have that for $\| \cdot \|$ being spectral or Frobenius norm, for
fixed $k$ as $n\to \infty$,
\[
    \frac{1}{\sqrt{n}}
    \big\| \sqrt{n} \, \mU_n \mD_n \mV_n^\trans - \mhX_n(k) \big\|
        \toas
        \big\|
            \diag \big(
                \mF_1(k),
                \mF_2(k),
                \ldots,
                \mF_{k \vee k_0}(k)
            \big) \big\|.
\]
Thus,
\begin{align}
    \frac{1}{n} \,
    \big\| \sqrt{n} \, \mU_n \mD_n \mV_n^\trans - \mhX_n(k) \big\|_\Frob^2
        &\toas
            \sum_{i=1}^{k \vee k_0}
                \big\| \mF_i(k) \big\|_\Frob^2 \\
\intertext{and} \label{E:spec2-loss-limit}
    \frac{1}{n} \,
    \big\| \sqrt{n} \, \mU_n \mD_n \mV_n^\trans - \mhX_n(k) \big\|_2^2
        &\toas
            \max_{1 \leq i \leq k \vee k_0}
                \big\| \mF_i(k) \big\|_2^2. 
\end{align}
Similar results can be gotten for other orthogonally-invariant norms.
A straightforward calculation shows
\[
    \begin{split}
    &\mF_i^\trans(k) \, \mF_i(k) \\
        &\,=
        \left(
        \begin{matrix}
            \mu_i 
            - 2 \varphi_i \mu_i^{1/2} \theta_i \bmu_i^{1/2}(k) 
            + \theta_i^2 \bmu_i(k) &
                - \varphi_i \bar \theta_i \mu_i^{1/2} \bmu_i^{1/2}(k)
                + \theta_i \bar \theta_i \bmu_i(k) \\
            - \varphi_i \bar \theta_i \mu_i^{1/2} \bmu_i^{1/2}(k)
            + \theta_i \bar \theta_i \bmu_i(k) &
                \bar \theta_i^2 \bmu_i(k)
        \end{matrix}
        \right),
    \end{split}
\]
so that
\begin{align*}
    \tr \big( \mF_i^\trans(k) \, \mF_i(k) \big)
        &= \mu_i 
           - 2 \varphi_i \theta_i \mu_i^{1/2} \bmu_i^{1/2} (k) 
           + \bmu_i(k), \\
    \det \big( \mF_i^\trans(k) \, \mF_i(k) \big)
        &= \bar \varphi_i^2 \bar \theta_i^2 \mu_i \bmu_i(k).
\end{align*}
When $\mu_i > \frac{\sigma^2}{\sqrt{\gamma}}$, we can use the identities
\(
    \varphi_i \theta_i \bmu_i^{1/2} \mu_i^{-1/2}
    =
    1
    -
    \frac{\sigma^4}{\gamma \mu_i^2}
\)
and
\(
    \bar \varphi_i^2 \bar \theta_i^2
    =
    \frac{\sigma^4}{\gamma \mu_i^2}
\)
to get
\begin{subequations}
\begin{align}\label{E:tr-det-F-above-a}
    \tr \big( \mF_i^\trans(k) \, \mF_i(k) \big)
        &=
            \begin{cases}
                \frac{\sigma^2}{\gamma \mu_i}
                \big(
                    3 \sigma^2 + (\gamma+1) \mu_i
                \big)
                    &\text{if $i \leq k$,} \\
                \mu_i
                    &\text{otherwise,}
            \end{cases} \\
    \det \big( \mF_i^\trans(k) \, \mF_i(k) \big)
        &=
            \begin{cases}
                \left(
                    \frac{\sigma^2}{\gamma \mu_i}
                \right)^2
                (\mu_i + \sigma^2)
                (\gamma \mu_i + \sigma^2)
                    &\text{if $i \leq k$} \\
                0
                    &\text{otherwise.}
            \end{cases}
\end{align}
\end{subequations}
When $\mu_i \leq \frac{\sigma^2}{\sqrt{\gamma}}$, we have
\begin{subequations}
\begin{align}
    \tr \big( \mF_i^\trans(k) \, \mF_i(k) \big)
        &=
            \begin{cases}
                \mu_i 
                +
                \sigma^2
                \left(
                    1 + \frac{1}{\sqrt{\gamma}}
                \right)^2
                    &\text{if $i \leq k$} \\
                \mu_i
                    &\text{otherwise,}
            \end{cases} \\
    \label{E:tr-det-F-below-b}
    \det \big( \mF_i^\trans(k) \, \mF_i(k) \big)
        &=
            \begin{cases}
                \mu_i \,
                \sigma^2
                \left(
                    1 + \frac{1}{\sqrt{\gamma}}
                \right)^2
                    &\text{if $i \leq k$} \\
                0
                    &\text{otherwise.}
            \end{cases}
\end{align}
\end{subequations}
We can use these expressions to compute
\begin{align}
    \| \mF_i(k) \|_\Frob^2
        &= \tr \big( \mF_i^\trans(k) \, \mF_i(k) \big), \\
    \label{E:F-spec2-formula}
    \begin{split}
    \| \mF_i(k) \|_2^2
        &= \frac{1}{2}
           \Big\{
                 \tr \big( \mF_i^\trans(k) \, \mF_i(k) \big) \\
                 &\qquad\qquad+
                 \sqrt{
                    \left[ 
                        \tr \big( \mF_i^\trans(k) \, \mF_i(k) \big) 
                    \right]^2
                    -
                    4
                    \det \big( \mF_i^\trans(k) \, \mF_i(k) \big) 
                 } \,
           \Big\}.
    \end{split}
\end{align}
The expression for the limit of 
\(
    \frac{1}{n} 
    \| \sqrt{n} \, \mU_n \mD_n \mV_n^\trans - \mhX_n(k) \|_2^2
\)
is fairly complicated.  In the Frobenius case, we have
\begin{proposition}\label{P:frob-loss-behavior}
    For fixed $k$ as $n \to \infty$, we have
    \begin{multline}
        \frac{1}{n}
        \| \sqrt{n} \, \mU_n \mD_n \mV_n^\trans - \mhX_n(k) \|_\Frob^2 \\
        \toas
        \sum_{i=1}^{k}
            \alpha_i \mu_i
        +
        \sum_{i=k+1}^{k_0}
            \mu_i
        +
        \sigma^2
        \left(
            1 + \frac{1}{\sqrt{\gamma}}
        \right)^2
        \cdot
        (k - k_0)_+,
    \end{multline}
    where
    \begin{equation}
        \alpha_i 
        =
        \begin{cases}
            \frac{\sigma^2}{\gamma \mu_i^2}
                    \big(
                        3 \sigma^2 + (\gamma+1) \mu_i
                    \big)
                &\text{if $\mu_i > \frac{\sigma^2}{\sqrt{\gamma}}$,} \\
            1 
            + 
            \frac{\sigma^2}{\mu_i}
            \left(
                1
                +
                \frac{1}{\sqrt{\gamma}}
            \right)^2
                &\text{otherwise.}
        \end{cases}
    \end{equation}
\end{proposition}

Figure~\ref{F:frobenius-loss-penalty} shows $\alpha_i$ as a function of
$\mu_i$.  It is beneficial to include the $i$th term when $\alpha_i < 1$,
or equivalently
\(
    \mu_i
    >
    \mu_\Frob^\ast,
\)
with
\begin{equation}\label{E:frob-cutoff}
    \mu_\Frob^\ast
    \equiv
    \sigma^2
    \left(
    \frac{1 + \gamma^{-1}}{2}
    +
    \sqrt{ 
        \left( \frac{1 + \gamma^{-1} }{2} \right)^2
        +
        \frac{3}{\gamma}
    }
    \right).
\end{equation}
This gives us the next Corollary.

\begin{figure}[tbh]
    \centering
    \includegraphics{frobenius-loss-penalty}
    \caption{
        \captiontitle{Frobenius loss penalty}
        Relative penalty for including the $i$th factor in the SVD 
        approximation of $\sqrt{n} \, \mU_n \mD_n \mV_n^\trans$, with
        respect to squared Frobenius loss.  When the $i$th factor
        has signal strengh $\mu_i$, the cost for excluding the $i$th term
        of the SVD is $\mu_i$, and the cost for including it is 
        $\alpha_i \cdot \mu_i$.  Here, we plot $\alpha_i$ as a function
        of $\mu_i$ for various aspect ratios $\gamma = \frac{n}{p}$.
        The units are chosen so that $\sigma^2 = 1$.
    }\label{F:frobenius-loss-penalty}
\end{figure}

\begin{corollary}
    As $n\to\infty$,
    \[
        k^\ast_\Frob
            \toas
                \max \left\{ i : \mu_i > \mu_\Frob^\ast \right\},
    \]
    provided no $\mu_k$ is exactly equal to $\mu_\Frob^\ast$.
\end{corollary}

\noindent
The theory in Chapter~\ref{C:svd-behavior} tells us that when 
$\mu_i > \frac{\sigma^2}{\sqrt{\gamma}}$, the $i$th signal term
is ``detectable'' in the sense that the $i$th sample singular value and singular vectors are correlated with the population quantities.  Proposition~\ref{P:frob-loss-behavior} tells us that when 
\(
    \mu_i 
    \in 
    \left(
        \frac{\sigma^2}{\sqrt{\gamma}}, \, 
        \mu_\Frob^\ast
    \right),
\)
the $i$th signal term is detectable, but it is \emph{not} helpful (in terms
of Frobenius norm) to include in the estimate of
\(
    \sqrt{n} \, \mU_n \mD_n \mV_n^\trans.
\)
Only when $\mu_i$ surpasses the inclusion threshold $\mu_\Frob^\ast$ is it
beneficial to include the $i$th term.

Figure~\ref{F:frobenius-loss-cutoff} shows the detection and inclusion 
thresholds as functions of $\gamma$.

\begin{figure}[hbt]
    \centering
    \includegraphics{frobenius-loss-cutoff}
    \caption{
        \captiontitle{Signal strength thresholds}
        Detection threshold $\gamma^{-1/2}$ and
        inclusion threshold
        $
            \mu_\Frob^\ast
            =
            \frac{1 + \gamma^{-1}}{2}
            +
            \sqrt{
                \left( \frac{1 + \gamma^{-1} }{2} \right)^2
                +
                \frac{3}{\gamma}
            }
        $
        plotted against the aspect ratio $\gamma = \frac{n}{p}$.  When
        the normalized signal strength $\frac{\mu_i}{\sigma^2}$ is above the 
        detection threshold, the $i$th sample SVD factors are correlated with
        the population factors.  With respect to Frobenius loss, when the 
        normalized signal strength is above the inclusion threshold, it is 
        beneficial to include the $i$th term in the SVD approximation of 
        $\sqrt{n} \, \mU_n \mD_n \mV_n^\trans$.
    }\label{F:frobenius-loss-cutoff}
\end{figure}

\clearpage

\section{Simulations}\label{S:intrinsic-loss-simulations}

We confirm the theory of the previous section with a Monte Carlo simulation.  We generate a matrix $\mX$ as follows:
\begin{enumerate}
    \item The concentration $\gamma$ is one of $\{ 0.25, 1.0, 4.0 \}$.
    \item The size of the matrix $s$ is one of $\{ 144, 400, 1600, 4900 \}$.
        We set the number of rows and columns in the matrix as
        $n = \sqrt{ s \, \gamma }$ and $p = \sqrt{s / \gamma}$, respectively.
        This ensures that $\gamma = \frac{n}{p}$ and $s = n \, p$.
    \item The noise level is set at $\sigma^2 = 1$.  The noise matrix $\mE$
        is an $n \times p$ matrix with iid $\Normal\left( 0, \, 1\right)$
        elements.
    \item The generative rank, $k_0$, is fixed at $5$.  The normalized
        factor strengths are set at
        \(
            (
                \mu_1, \,
                \mu_2, \,
                \mu_3, \,
                \mu_4, \,
                \mu_5
            )
            =
            (
                4 \mu_\Frob^\ast, \,
                2 \mu_\Frob^\ast, \,
                  \mu_\Frob^\ast, \,
                \frac{1}{2} \mu_\Frob^\ast, \,
                \frac{1}{4} \mu_\Frob^\ast
            ),
        \)
        and the factor strength matrix is
        \(
            \mD 
            = 
            \diag\left( 
                \mu_1^{1/2},
                \mu_2^{1/2},
                \ldots, 
                \mu_5^{1/2}
            \right).
        \)
    \item The left and right factor matrices $\mU$ and $\mV$ are of sizes
        $n \times 5$ and $p \times 5$, respectively.  We choose these matrices
        uniformly at random over the Stiefel manifold according
        to Algorithm~\ref{A:random-steifel} in Appendix~\ref{A:projections}.
    \item We set $\mX = \sqrt{n} \, \mU \mD \mV^\trans + \mE$ and let
        $\mhX(k)$ be the SVD of $\mX$ truncated to $k$ terms.
\end{enumerate}
After generating $\mX$, we compute the squared Frobenius loss
\(
    \frac{1}{n} \| \sqrt{n}\mU \mD \mV^\trans - \mhX(k) \|_F^2
\)
and the squared spectral loss
\(
    \frac{1}{n} \| \sqrt{n}\mU \mD \mV^\trans - \mhX(k) \|_2^2
\)
as functions of the rank, $k$.  In 
Figures~\ref{F:frob2-loss-sim}~and~\ref{F:spec2-loss-sim}, we plot the results 
over 500 replicates.  For the Frobenius case, we should expect the loss to
decrease until $k=2$, stay flat at $k=3$, and then increase thereafter.  This 
is confirmed by the simulations.  The spectral norm behaves similarly to the 
Frobenius norm.


\begin{figure}[tbh]
    \centering
    \includegraphics{frob2-loss-sim}
    \caption{
        \captiontitle{Simulated Frobenius loss}
        Squared Frobenius loss
        $
            \frac{1}{n} \| \sqrt{n}\mU \mD \mV^\trans - \mhX(k) \|_F^2
        $
        as a function of the rank, $k$, for
        $\mX$ generated according the procedure described
        in Section~\ref{S:intrinsic-loss-simulations} and $\mhX(k)$ equal
        to the SVD of $\mX$ truncated to $k$ terms.  The concentration 
        $\gamma = \frac{n}{p}$ is one of $\{ 0.25, 1.0, 4.0 \}$ and the
        size $s = n p$ is one of $\{ 144, 400, 1600, 4900 \}$.  
        The solid lines show the means over $500$ replicates with the error 
        bars showing one standard deviation.  The dashed lines show the 
        predictions from Proposition~\ref{P:frob-loss-behavior}.  We can see 
        that as the size increases, the simulation agrees more and more with 
        the theory.
    }\label{F:frob2-loss-sim}
\end{figure}

\begin{figure}[hbt]
    \centering
    \includegraphics{spec2-loss-sim}
    \caption{
        \captiontitle{Simulated spectral loss}
        Squared spectral loss
        $
            \frac{1}{n} \| \sqrt{n}\mU \mD \mV^\trans - \mhX(k) \|_2^2
        $
        as a function of the rank, $k$, with $\mX$ and $\mhX(k)$ as in
        Figure~\ref{F:frob2-loss-sim}.  Sample size $s = n p$ is shown in the 
        columns and concentration $\gamma = \frac{n}{p}$ is shown in the rows.
        Solid lines show the means
        over $500$ replicates with the error bars showing one standard 
        deviation; the dashed lines show the predictions from 
        Section~\ref{S:loss-behavior}, specifically from 
        equations~\eqref{E:spec2-loss-limit}, 
        (\ref{E:tr-det-F-above-a}--\ref{E:tr-det-F-below-b}), 
        and~\eqref{E:F-spec2-formula}.  The simulations agrees quite well
        with the theory, especially for large sample sizes.
    }\label{F:spec2-loss-sim}
\end{figure}

\clearpage

\section{Relation to the scree plot}\label{S:intrinsic-rank-scree-plot}

Cattell's scree plot~\cite{cattell1966scree} is a popular device for choosing
the truncation rank in principal component analysis. One plots the square of
the singular value, $\hat d^2_k$, against the component number, $k$. Typically
such a plot exhibits an ``elbow'', where the slope changes noticeably. This is
the point at which Cattell recommends truncating the SVD of the matrix.

In some circumstances, the elbow is close to the Frobenius and spectral loss
minimizers, $k_\Frob^\ast$ and $k_2^\ast$. In Figure~\ref{F:scree-elbow}, we
generate a matrix
\(
    \mX = \sqrt{n} \, \mU \mD \mV^\trans + \mE \in \reals^{n \times p},
\)
with $n = p = 100$, such that elements of $\mE$ are iid $\Normal( 0, \, 1)$. 
In the first column, we set
\(
    \mD^2 = \diag( 5.0, \, 4.75, \, 4.5, \, \ldots, \, 0.5, \, 0.25 ).
\)
The figure shows the scree plot along with
\(
    \| \sqrt{n} \, \mU \mD \mV^\trans - \mhX(k) \|^2
\) 
for Frobenius and spectral norms, where $\mhX(k)$ is the SVD of $\mX$
truncated to $k$ terms. There is substantial ambiguity in determining the
location of the most pronounced elbow. Despite this subtlety, there is indeed 
an elbow relatively close to $k_\Frob^\ast$ and $k_2^\ast$, the minimizers of 
the two loss functions.

We can easily manipulate the simulation to get a scree plot with a more 
pronounced elbow in the wrong place.  In the second column of Figure~\ref{F:scree-elbow}, we augment the factor strength matrix with three additional large values, so that
\(
    \mD^2 
    = 
    \diag( 
        20.0, \, 15.0, \, 10.0, \,
        5.0, \, 4.75, \, 4.5, \, \ldots, \, 0.5, \, 0.25 
    ).
\)
With this modification, there is a clear elbow at $k=4$.  However, compared to 
the optimal values, truncating the SVD at $k=4$ gives about a 25\% worse error 
with respect to squared Frobenius loss and about 50\% worse with respect to 
squared spectral loss.

In general, we cannot make any assurances about how close the elbow is to 
$k_\Frob^\ast$ or $k_2^\ast$.  Through a simulation study, 
Jackson~\cite{jackson1993stopping} provides evidence that if the latent 
factors are strong enough to be easily distinguished from noise, then the 
elbow is a reasonable estimate of the loss minimizers (he does not actually compute the loss behavior, but this seems likely).  However, when there 
are both well-separated factors and factors near the critical strength level 
$\mu_\Frob^\ast$, the second example here illustrates that the elbow might be 
a poor estimate.

\begin{figure}[ht]
    \begin{center}
        \begin{minipage}{0.49\textwidth}
            \begin{center}
                \includegraphics{scree-elbow-right}
            \end{center}
        \end{minipage}
        \begin{minipage}{0.49\textwidth}
            \begin{center}
                \includegraphics{scree-elbow-left}
            \end{center}
        \end{minipage}
        \caption{
            \captiontitle{Scree plots and loss functions}
            We generate a matrix as 
            $\mX = \sqrt{n} \, \mU \mD \mV^\trans + \mE$ 
            and set $\mhX(k)$ equal to the SVD of $\mX$ truncated to $k$ 
            terms.  The left and right columns show different choices of
            $\mD$, described in the text.  The top row contains scree plots, 
            where the square of the $k$th singular value of $\mX$,
            $\hat d_k^2$, is plotted against 
            component number, $k$, with units are chosen so that 
            $\sum_{k} \hat d_k^2 = 1$.  The next two rows show
            $\| \sqrt{n} \, \mU \mD \mV^\trans - \mhX(k) \|_\Frob^2$ and
            $\| \sqrt{n} \, \mU \mD \mV^\trans - \mhX(k) \|_2^2$, as functions
            of the rank, $k$ with units chosen so that the minimum value is
            $1.0$.  Dashed lines show the elbow of the scree plot and the
            minimizers of the two loss functions.  
            The elbow of the scree plot (which is fit by eye)
            gives a reasonable estimate of the loss minimizers in the first 
            column, but not in the second.
        }\label{F:scree-elbow}
    \end{center}
\end{figure}

\clearpage

\section{Extensions}\label{S:intrinsic-rank-extensions}

We have focused specifically on truncating the singular value decomposition
because that is the case for which the theory has been most developed. We have
also focused specifically on Frobenius and spectral losses. One could easily
extend our work to look at the nuclear norm loss (sum of singular values)
\cite{fazel2002mrm} or Stein's loss \cite{james1961ewq}. Alternatively, it is
not hard to adapt our analysis to study forms like $\| \mhU \mhD(k)^2
\mhU^\trans - \mU \mD^2 \mU^\trans \|$. For matrix decompositions beyond the
SVD, extension of our work is more difficult, mainly because very little
scholarship has been devoted to their theoretical properties.

We have not examined shrinking the singular values at all, but in some 
situations this may be beneficial.  For example, the Frobenius norm of the $\mF_i$ matrix from Section~\ref{S:loss-behavior}, which is involved in the Frobenius loss
\(
    \| \sqrt{n} \, \mU \mD \mV^\trans - \mhX(k) \|_F^2,
\)
converges to
\[
    \mu_i - 2 \varphi_i \theta_i \mu_i^{1/2} \bmu_i^{1/2} + \bmu_i
\]
whenever $k \geq i$.  Recall that $\bmu_i$ is the almost-sure limit of
the $\hat d_i^2$, the square of the $i$th singular value of $\frac{1}{\sqrt{n}} \mX$.  If we shrink the $i$th singular value, replacing
$\hat d_i$ with $f(\hat d_i)$ for continuous $f(\cdot)$, then the Frobenius penalty for including the $i$th term converges to
\[
    \mu_i 
    - 2 \varphi_i \theta_i \mu_i^{1/2} f( \bmu_i^{1/2} ) 
    + f^2( \bmu_i^{1/2} ).
\]
This quantity is minimized when
\(
    f( \bmu_i^{1/2}) 
        = \mu_i^{1/2} \varphi_i \theta_i
        = \bmu_i^{-1/2} 
          \left( \mu_i - \frac{\sigma^4}{ \gamma \mu_i } \right).               
\)
After some algebra, the optimal $f$ takes the form
\begin{equation}
    f( \hat d_i )
        =
        \begin{cases}
        \left(
            \hat d_i^2
            -
            2
            \big(
                1 + \frac{1}{\gamma}
            \big)
            \sigma^2
            +
            \big(
                1 - \frac{1}{\gamma}
            \big)^2
            \frac{\sigma^4}
                 {\hat d_i^2}
        \right)^{1/2}
            &\text{when $\hat d_i 
                         > 
                         \sigma 
                         \left(
                            1 + \frac{1}{\sqrt{\gamma}}
                         \right)$,} \\
        0 
            &\text{otherwise.}
        \end{cases}
\end{equation}
With this shrinkage, it is always beneficial (in terms of Frobenius norm) to include the $i$th term.

\section{Summary and future work}\label{S:intrinsic-rank-summary}

We have described a plausible model for data generated by a small number of
latent factors corrupted by additive white noise. With this model in mind, we
have motivated two loss functions, the squared Frobenius norm and squared
spectral norm, as measuring average- or worst-case quadratic error over the
class of all bilinear statistics. These loss functions in turn determine the
intrinsic ranks $k_\Frob^\ast$ and $k_2^\ast$. We have shown how the losses
and the ranks behave for the truncated SVD, both theoretically and through
simulation. Finally, we have explored the relation between intrinsic rank and
the scree plot, and then discussed some extensions.

We did not describe a way to \emph{estimate} the error from truncating an SVD,
nor did we propose a practical method for choosing $k$. For now, our hope is
that this work is useful in developing intuition for how the SVD behaves and
that it provides a suitable starting point for designing and evaluating such
procedures. In later chapters, we explicitly discuss estimation and rank
selection.

introduction.tex
\chapter{Introduction}

Cross-validation (CV) is a popular method for model selection.  It works by estimating the prediction error of each model under consideration and then choosing the model with the best performance.  In unsupervised contexts, though, there is no clear notion as to what exactly ``prediction error'' is.  Therefore, it is difficult to employ cross-validation for model selection in unsupervised or exploratory contexts.  In this thesis, we take a look at some extensions of cross-validation to unsupervised learning.  We focus specifically on the problem of choosing how many components to keep for principal component analysis (PCA), but many of the concepts we introduce are more broadly applicable.

Before we can do anything, we need a solid theoretical foundation.  To this end, Chapter~\ref{C:multivariate-background} gives a survey of relevant results from multivariate statistics and random matrix theory.  Then, Chapter~\ref{C:svd-behavior} derives the behavior of the singular value decomposition (SVD) for ``signal-plus-noise'' matrix models.  These two chapters are complemented by Appendices~\ref{A:projections}~and~\ref{A:weighted-sums}, which collect properties of random orthogonal matrices and give some limit theorems for weighted sums of random variables.  Collectively, this work provides the groundwork for the rest of the thesis.

In Chapter~\ref{C:intrinsic-rank}, we introduce the latent factor model.  This is a generative model for signal-plus-noise matrix data that expands the setup of PCA to include correlated factor loadings.  We motivate loss functions for estimating the signal part, and then show how the SVD performs with respect to these criteria.  

The next chapter (Chapter~\ref{C:cvul}) focuses on cross-validation strategies.  It covers both Wold-style ``speckled'' hold-outs as well as Gabriel-style ``blocked'' hold-outs.  We define model error and prediction error for the latent factor model, and present the two cross-validation methods as estimators of prediction error.  The chapter includes a comparison of CV methods with parametric model-selection procedures, showing through simulation that CV is much more robust to violations in model assumptions.  In situations where parametric assumptions are unreasonable, cross-validation proves to be an attractive method for model selection.

Chapter~\ref{C:optimal-leave-out}, the final chapter, contains a theoretical analysis of Gabriel-style cross-validation for the SVD, also known as bi-cross-validation (BCV).  This chapter shows that BCV is in general a biased estimator of model error, with an explicit expression for the bias.  Despite this bias, though, the procedure can still be used successfully for model selection, provided the leave-out sizes are chosen appropriately.  The chapter shows how to choose the leave-out sizes and proves a weak form of consistency.

Cross-validation is a valuable and flexible procedure for model selection.  Through theory and simulation, this thesis demonstrates the applicability and utility of cross-validation as applied to principal component analysis.  Many of the ideas in the following chapters generalize to other unsupervised learning procedures.  This thesis shows that cross-validation can successfully be used for model selection in a variety of contexts.

lowrank.tex

\chapter[Behavior of the SVD]{Behavior of the SVD in low-rank plus noise models}\label{C:svd-behavior}

\section{Introduction}

Many modern data sets involve simultaneous measurements of a large number of
variables. Some financial applications, such as portfolio selection, involve
looking at the market prices of hundreds or thousands of stocks and their
evolution over time~\cite{markowitz1952ps}. Microarray studies
involve measuring the expression levels of thousands of
genes simultaneously~\cite{schena1995qmg}.  Text processing involves counting
the appearances of thousands of words on thousands of documents~\cite{manning1999fsn}.  In all of these
applications, it is natural to suppose that even though the data are
high-dimensional, their dynamics are driven by a relatively small number of
latent factors.

Under the hypothesis that one ore more latent factors explain the behavior
of a data set, principal component analysis (PCA) \cite{jolliffe2002pca} is a
popular method for estimating these latent factors.  When the dimensionality
of the data is small relative to the sample size, Anderson's 1963 paper
\cite{anderson1963atp} gives a complete treatment of how the procedure
behaves.  Unfortunately, his results do not apply when the sample size is
comparable to the dimensionality.

A further complication with many modern data sets is that it is no longer
appropriate to assume the observations are iid. Also, sometimes it is
difficult to distinguish between ``observation'' and ``variable''. We call such
data ``transposable''. A microarray study involving measurements of $p$ genes
for $n$ different patients can be considered transposable: we can either treat
each gene as a measurement of the patient, or we can treat each patient as a
measurement of the gene. There are correlations both between the genes
\emph{and} between the patients, so in fact both interpretations are relevant~\cite{efron2008smi}.

One can study latent factor models in a transpose-agnostic way by considering
generative models of the form $\mX = \mU \mD \mV^\trans + \mE$. Here, $\mX$ is
the $n \times p$ observed data matrix. The unobserved row and column factors
are given by $\mU$ and $\mV$, respectively, matrices with $k$ orthonormal
columns each, and $k \ll \min(n,p)$. The strengths of the factors are given
in the $k\times k$ diagonal matrix $\mD$, and $\mE$ is a matrix of noise. A
natural estimator for the latent factor term $\mU \mD \mV^\trans$ can be
constructed by truncating the singular value decomposition (SVD)
\cite{golub1996mc} of $\mX$. The goal of this paper is to study the behavior
of the SVD when $n$ and $p$ both tend to infinity, with their ratio tending to
a nonzero constant.

In an upcoming paper, Onatski~\cite{onatski2009} gives a thorough
treatment of latent factor models. He assumes that the elements of $\mE$ are iid
Gaussians, that $\sqrt{n} ( \mU^\trans \mU - \mI_k) $ tends to a multivariate
Gaussian distribution, and that $\mV$ and $\mD$ are both nonrandom. The
contributions of this chapter are twofold. First, we work under a
transpose-agnostic generative model that allows randomness in all three of
$\mU$, $\mD$, and $\mV$. Second, we give a more complete picture of the
almost-sure limits of the sample singular vectors, taking into account the
signs of the dot products between the population and sample vectors.

We describe the main results in Section~\ref{S:lowrank-assumptions-results}.
Sections~\ref{S:lowrank-preliminaries}--\ref{S:lowrank-gamma-lt-1} are dedicated to proving the two main theorems.  Finaly, we discuss related work and extensions in Section~\ref{S:lowrank-related-extensions}.  We owe a substantial debt to Onatski's work.  Although most of the details below are different, the general outline and the main points of the argument are the same.

\section{Assumptions, notation, and main results}\label{S:lowrank-assumptions-results}

Here we make explicit what the model and assumptions are, and we present our
main results.

\subsection{Assumptions and notation}

We will work sequences of matrices indexed by $n$, with
\begin{equation}
    \mX_n = \sqrt{n} \, \mU_n \mD_n \mV_n^\trans + \mE_n.
\end{equation}
We denote by $\sqrt{n} \, \mU_n \mD_n \mV_n^\trans$ the ``signal'' part of the
matrix $\mX_n$ and $\mE_n$ the ``noise'' part. We will often refer to $\mU_n$
and $\mV_n$ as the left and right factors of $\mX_n$, and the matrix $\mD_n$
will be called the matrix of normalized factor strengths. The first two
assumptions concern the signal part:

\begin{assumption}\label{A:factors}
    The factors $\mU_n$ and $\mV_n$ are random matrices of dimensions
    $n \times k$ and $p \times k$, respectively, normalized so that
    $\mU_n^\trans \mU_n =  \mV_n^\trans \mV_n = \mI_k$. 
    The number of factors, $k$, is a fixed constant.  The aspect ratio
    satisfies $\frac{n}{p} = \gamma + \oh\left( \frac{1}{\sqrt{n}} \right)$
    for a fixed nonzero constant $\gamma \in (0,\infty)$.
\end{assumption}
\noindent

\begin{assumption}\label{A:sizes}
    The matrix of factor strengths, $\mD_n$, is of size $k\times k$ and 
    diagonal, with 
    \(
        \mD_n = \diag \left( d_{n,1}, d_{n,2}, \ldots, d_{n,k} \right)
    \)
    and
    $d_{n,1} > d_{n,2} > \cdots > d_{n,k} > 0$.  The matrix $\mD_n$ converges
    almost surely to a deterministic matrix 
    $\mD = \diag( \mu_1^{1/2}, \mu_2^{1/2}, \ldots, \mu_k^{1/2})$ with
    $\mu_1 > \mu_2 > \cdots > \mu_k > 0$. Moreover, the vector
    \(
        \sqrt{n} ( d_{n,1}^2 - \mu_1, d_{n,1}^2 - \mu_2, 
                   \ldots, 
                   d_{n,K}^2 - \mu_k )
    \)
    converges in distribution to a mean-zero multivariate normal with 
    covariance matrix $\mSigma$ having entries $\Sigma_{ij} = \sigma_{ij}$
    (possibly degenerate).
\end{assumption}
\noindent
The next assumption concerns the noise part:

\begin{assumption}\label{A:noise}
    The noise matrix $\mE_n$ is an $n\times p$ matrix with entries 
    $E_{n,ij}$ independent $\Normal(0,\, \sigma^2)$ random variables, also
    independent of $\mU_n$, $\mD_n$, and $\mV_n$.
\end{assumption}
\noindent

For analyzing the SVD of $\mX_n$, we need to introduce some more notation. We denote the columns of $\mU_n$ and
$\mV_n$ by $\vu_{n,1}, \vu_{n,2}, \ldots, \vu_{n,k}$ and $\vv_{n,1},
\vv_{n,2}, \ldots, \vv_{n,k}$, respectively. We let 
\(
    \sqrt{n} \, \mhU_n \mhD_n \mhV_n^\trans
\)
be the singular value decomposition of
$\mX_n$ truncated to $k$ terms, where $\mhD_n = \diag( \hat \mu_{n,1}^{1/2},
\hat \mu_{n,2}^{1/2}, \ldots, \hat \mu_{n,k}^{1/2})$ and the columns of $\mhU$ and $\mhV$ are given
by $\vhu_{n,1}, \vhu_{n,2} \ldots, \vhu_{n,k}$ and $\vhv_{n,1}, \vhv_{n,2},
\ldots, \vhv_{n,k}$, respectively.


\subsection{Main results}

We are now in a position to say what the results are.  There are two main
theorems, one concerning the sample singular values and the other concerning
the sample singular vectors.  First we give the result about the singular
values.
\begin{theorem}\label{T:spiked-eigenvalue-limits}
    Under Assumptions~\ref{A:factors}~--~\ref{A:noise},
    the vector $\hat \vmu_n = (\hmu_{n,1}, \hmu_{n,2}, \ldots, \hmu_{n,k})$
    converges almost surely to 
    $\vbmu = (\bmu_1, \bmu_2, \ldots, \bmu_k)$,
    where
    \begin{equation}
        \bmu_i
        =
        \begin{cases}
            \left( \mu_i + \sigma^2 \right)
            \left( 1 + \frac{\sigma^2}{\gamma \mu_i} \right)
                &\text{when $\mu_i > \frac{\sigma^2}{\sqrt{\gamma}}$}, \\
            \sigma^2 \left( 1 + \frac{1}{\sqrt{\gamma}} \right)^2
                &\text{otherwise.}
        \end{cases}
    \end{equation}
    Moveover, 
    \(
        \sqrt{n} (\vhmu - \vbmu)
    \)
    converges in distribution to a (possibly degenerate) multivariate normal
    with covariance matrix $\mbSigma$ whose $ij$ element is given by
    \begin{equation}
        \bsigma_{ij}
        \equiv
        \begin{cases}
            \begin{aligned}
                \sigma_{ij}
                &\left(
                    1 - \frac{\sigma^4}{\gamma \mu_i^2}
                \right)
                \left(
                    1 - \frac{\sigma^4}{\gamma \mu_j^2}
                \right)
                \\
                &+ \delta_{ij}
                2
                \sigma^2 \,
                \bigg(
                    2 \mu_i + (1 + \gamma^{-1}) \, \sigma^2
                \bigg)
                \bigg(
                    1 - \frac{ \sigma^4 }{ \gamma  \mu_i^2 }
                \bigg)
            \end{aligned}
                &\text{when $\mu_i, \, \mu_j >
                           \frac{\sigma^2}{\sqrt{\gamma}}$,} \\
            0
                &\text{otherwise.}
        \end{cases}
    \end{equation}
    When $\sigma_{ii} = 2 \mu_i^2$, 
    and $\mu_i > \frac{\sigma^2}{\sqrt{\gamma}}$, the variance of the $i$th
    component simplifies to
    \(
        \bsigma_{ii} 
        = 
        2 (\mu_i + \sigma^2)^2 
        \left( 1 - \frac{\sigma^4}{\gamma \mu_i^2} \right).
    \)
\end{theorem}

Next, we give the result for the singular vectors:
\begin{theorem}\label{T:spiked-eigenvector-limits}
    Suppose Assumptions~\ref{A:factors}~--~\ref{A:noise} hold.  Then the
    $k\times k$ matrix $\mTheta_n \equiv \mV_n^\trans \mhV_n$ converges almost 
    surely to a matrix 
    $\mTheta = \diag(\theta_1, \theta_2, \ldots, \theta_k)$, where
    \begin{equation}
        \theta_i^2
        =
        \begin{cases}
            \left( 1 - \frac{\sigma^4}{ \gamma \mu_i^2} \right)
            \left( 1 + \frac{\sigma^2}{ \gamma \mu_i  } \right)^{-1}
            &\text{when $\mu_i > \frac{\sigma^2}{\sqrt{\gamma}}$,} \\
            0
            &\text{otherwise.}
        \end{cases}
    \end{equation}
    Also, the $k\times k$ matrix $\mPhi_n \equiv \mU_n^\trans \mhU_n$
    converges almost surely to a matrix
    $\mPhi = \diag(\varphi_1, \varphi_2, \ldots, \varphi_k)$, where    
    \begin{equation}
        \varphi_i^2
        =
        \begin{cases}
            \left( 1 - \frac{\sigma^4}{ \gamma \mu_i^2} \right)
            \left( 1 + \frac{\sigma^2}{ \mu_i  } \right)^{-1}
            &\text{when $\mu_i > \frac{\sigma^2}{\sqrt{\gamma}}$,} \\
            0
            &\text{otherwise.}
        \end{cases}
    \end{equation}
    Moreover, $\theta_i$ and $\varphi_i$ almost surely have the same
    sign.
\end{theorem}


\subsection{Notes}

Some discussion of the assumptions and the results are in order:

\begin{enumerate}
    
\item
Assumptions~\ref{A:factors}~and~\ref{A:sizes} are simpler than the assumptions given in many other papers while still being quite general.  For example, Paul's ``spiked'' covariance model has data of the form
\[
    \mX = \mZ \mXi^\trans + \mE,
\]
where $\mXi$ is an $p \times k$ matrix of factors and $\mZ$ is an $n\times k$ matrix of factor loadings whose rows are iid multivariate $\Normal(0,\, \mC)$ random variables for covariance matrx $\mC$ having eigen-decomposition $\mC = \mQ \mLambda \mQ^\trans$.  Letting $\mZ = \sqrt{n} \mhP \mhLambda^{1/2} \mhQ^\trans$ be the SVD of $\mZ$, Anderson's results \cite{anderson1963atp} give us that $\mhLambda$ and $\mhQ$ converge almost surely to $\mLambda$ and $\mQ$, respectively, and that the diagonal elements of $\sqrt{n} (\mhLambda - \mLambda)$ converge to a mean-zero multivariate normal whenever $\mC$ has no repeated eigenvalues.  If we define $\mU = \mhP$, $\mD = \mhLambda^{1/2}$, and $\mV = \mXi \mhQ$, then $\mX = \sqrt{n} \, \mU \mD \mV^\trans + \mE$, where the factors satisfy Assumptions~\ref{A:factors}~and~\ref{A:sizes}.  Dropping the normality assumption on the rows of $\mZ$ poses no problem.  Moreover, we can suppose instead of iid that the rows of $\mZ$ are a martingale difference array with well-behaved low-order moments and still perform a transformation of the variables to get factors of the form we need for Theorems~\ref{T:spiked-eigenvalue-limits}~and~\ref{T:spiked-eigenvector-limits}.

\item
There is a sign-indeterminancy in the sample and population singular vectors. We choose them arbitrarily.

\item
If instead of almost-sure convergence, $\mD_n$ converges in probability to $\mD$,
then the theorems still hold with $\vhmu_n$, $\mTheta_n$, and $\mPhi_n$ converging
in probability.

\item
The assumption that
\(
    \sqrt{n}
    \left(
        d_1^2 - \mu_1,
        d_2^2 - \mu_2,
        \ldots
        d_k^2 - \mu_k
    \right)
\)
converges weakly is only necessary for determining second-order
behavior; the first order results still hold without this assumption.  If
the limiting distribution of the vector of factor strengths
\(
    \sqrt{n}
    \left(
        d_1^2 - \mu_1,
        d_2^2 - \mu_2,
        \ldots
        d_k^2 - \mu_k
    \right)
\)
is non-normal, one can still get at the second-order behavior of the SVD 
through a small adaptation of the proof.

\item 
Most of the results in Theorems~\ref{T:spiked-eigenvalue-limits}~and~\ref{T:spiked-eigenvector-limits} can be gotten from Onatski's results \cite{onatski2009}.   However, Onatski does not show that $\sqrt{n} (\hmu_i - \bmu_i) \toP 0$ when $\mu_i$ is below the critical threshold.  Furthermore, Onatski proves convergence in probability, not almost sure convergence.  Lastly, Onatski does not get at the joint behavior between $\mTheta_n$ and $\mPhi_n$.

\end{enumerate}

\section{Preliminaries}\label{S:lowrank-preliminaries}

Without loss of generality we will assume that $\sigma^2 = 1$.  Until Section~\ref{S:lowrank-gamma-lt-1} , we will also assume that $\gamma \geq 1$.

\subsection{Change of basis}

A convenient choice of basis will make it easier to study the SVD of $\mX_n$.  Define $\mU_{n,1} = \mU_n$, and choose $\mU_{n,2}$ so that
\(
    \left( 
    \begin{matrix}
        \mU_{n,1} & \mU_{n,2}
    \end{matrix}
    \right)
\) 
is an orthogonal matrix. Similarly, put $\mV_{n,1} = \mV_n$ and choose
$\mV_{n,2}$ so that
\(
    \left(
    \begin{matrix}
        \mV_{n,1} & \mV_{n,2}
    \end{matrix}
    \right)
\)
is orthogonal.  If we define
\(
    \mtE_{n,ij} = \mU_{n,i}^\trans \mE_n \mV_{n,j}
\)
and
\(
    \mX_{n,ij} = \mU_{n,i}^\trans \mX_n \mV_{n,j},
\)
then in block form,
\[
    \left( 
    \begin{matrix}
        \mU_{n,1}^\trans \\
        \mU_{n,2}^\trans
    \end{matrix}
    \right)
    \mX_n
    \left(
    \begin{matrix}
        \mV_{n,1} & \mV_{n,2}
    \end{matrix}
    \right)
    =
    \left(
    \begin{matrix}
        \sqrt{n} \mD_n + \mtE_{n,11} & \mtE_{n,12} \\
        \mtE_{n,21}                  & \mtE_{n,22}
    \end{matrix}
    \right).
\]
Because Gaussian white noise is orthogonally invariant, the
matrices $\mtE_{n,ij}$ are all independent with iid $\Normal(0, \, 1)$ elements.  Let
\begin{equation}
    \mtE_{n,22}
    =
    \sqrt{n}
    \left(
    \begin{matrix}
        \mO_{n,1} & \mO_{n,2}
    \end{matrix}
    \right)
    \left(
    \begin{matrix}
        \mLambda_n^{1/2} \\
        0
    \end{matrix}
    \right)
    \mP_{n}^\trans
\end{equation}
be the SVD of $\mtE_{n,22}$, with
\(
    \mLambda_n
    =
    \diag \left(
        \lambda_{n,1}, 
        \lambda_{n,2}, 
        \ldots
        \lambda_{n,p-k}
    \right)
\)
.  Note that 
\(
    \mtE_{n,22}^\trans \mtE_{n,22}
    \sim
    \Wishart_{p-k} \left( n-k, \mI_{p-k} \right).
\)
Define
\begin{align}
    \mtX_n
        &=
            \left(
            \begin{matrix}
                \mI_k & 0 \\
                0  & \mO_{n,1}^\trans \\
                0  & \mO_{n,2}^\trans
            \end{matrix}
            \right)
            \left(
            \begin{matrix}
                \sqrt{n} \mD_n + \mtE_{n,11} & \mtE_{n,12} \\
                \mtE_{n,21}                  & \mtE_{n,22}
            \end{matrix}
            \right)
            \left(
            \begin{matrix}
                \mI_k & 0 \\
                0     & \mP_n
            \end{matrix}
            \right) \notag \\
        &=
            \left(
            \begin{matrix}
                \sqrt{n} \mD_n + \mE_{n,11} & \mE_{n,12} \\
                \mE_{n,21}                  & \sqrt{n} \mLambda^{1/2}_n \\
                \mE_{n,31}                  & 0
            \end{matrix}
            \right),
\end{align}
where 
$\mE_{n,11} = \mtE_{n,11}$, 
$\mE_{n,12} = \mtE_{n,12} \mP_n$,
$\mE_{n,21} = \mO_{n,1}^\trans \mtE_{n,21}$, and
$\mE_{n,31} = \mO_{n,2}^\trans \mtE_{n,31}$.
Let
\(
    \mtU_n \mtD_n \mtV_n
\)
be the SVD of $\mtX_n$, truncated to $k$ terms.  Lastly, put the left and right singular vectors in block form as
\begin{equation}
    \mtU_n
    =
    \left(
    \begin{matrix}
        \mtU_{n,1} \\
        \mtU_{n,2}
    \end{matrix}
    \right)
\end{equation}
and
\begin{equation}
    \mtV_n
    =
    \left(
    \begin{matrix}
        \mtV_{n,1} \\
        \mtV_{n,2}
    \end{matrix}
    \right),
\end{equation}
where $\mtU_{n,1}$ and $\mtV_{n,1}$ both $k\times k$ matrices.

We have gotten to $\mtX_n$ via an orthogonal change of basis applied to $\mX_n$.  By carefully choosing this basis, we have assured that:
\begin{enumerate}
    \item The blocks of $\mtX_n$ are all independent.
    \item The elements of the matrices $\mE_{n,ij}$ are 
        iid $\Normal\left( 0, 1 \right)$.
    \item The elements 
        $\{ n \lambda_{n,1}, n \lambda_{n,2}, \ldots, 
            n \lambda_{n,p-k} \}$ are eigenvalues from
        a white Wishart matrix with $n-k$ degrees of freedom.
    \item $\mtX_n$ and $\mX_n$ have the same singular values.  This implies
        that $\mhD_n = \mtD_n$.
    \item The left singular vectors of $\mX_n$ can be recovered from the
        left singular vectors of $\mtX_n$ via multiplication by an orthogonal
        matrix.  The same holds for the right singular vectors.
    \item The dot product matrix $\mU_n^\trans \mhU_n$ is equal to
        $\mtU_{n,1}$.
        Similarly, $\mV_n^\trans \mhV_n = \mtV_{n,1}$.
\end{enumerate}
This simplified form of the problem makes it much easier to analyze the 
SVD of $\mX_n$.


\section{The secular equation}\label{S:secular-equation}

We set $\mS_n = \frac{1}{n} = \mtX_n^\trans \mtX_n$.  The eigenvalues and eigenvectors of $\mS_n$ are the squares of the singular values of $\frac{1}{\sqrt{n}} \mtX_n$ and its right singular vectors, respectively.  In block form, we have
\begin{equation}
    \mS_n
    =
    \left(
    \begin{matrix}
        \mS_{n,11} & \mS_{n,12} \\
        \mS_{n,21} & \mS_{n,22}
    \end{matrix}
    \right),
\end{equation}
where
\begin{subequations}
\begin{align}
    \begin{split}
    \mS_{n,11}
        &=
            \mD_n^2 
            + 
            \frac{1}{\sqrt{n}} 
            \left( 
                \mD_n \mE_{n,11} + \mE_{n,11}^\trans \mD_n
            \right) \\
            &\phantom{= \mD_n^2} +
            \frac{1}{n}
            \left(
                \mE_{n,11}^\trans \mE_{n,11}
                +
                \mE_{n,21}^\trans \mE_{n,21}
                +
                \mE_{n,31}^\trans \mE_{n,31}
            \right),
    \end{split}
\end{align}
and
\begin{align}
    \mS_{n,12}
        &=
            \frac{1}{\sqrt{n}}
            \left(
                \mD_n \mE_{n,12} + \mE_{n,21}^\trans \mLambda_n^{1/2}
            \right)
            +
            \frac{1}{n}
            \mE_{n,11}^\trans \mE_{n,12}, \\
    \mS_{n,21}
        &=
            \frac{1}{\sqrt{n}}
            \left(
                \mE_{n,12}^\trans \mD_n
                +
                \mLambda_n^{1/2} \mE_{n,21}
            \right)
            +
            \frac{1}{n}
            \mE_{n,12}^\trans \mE_{n,11}, \\
    \mS_{n,22}
        &=
            \mLambda_n
            +
            \frac{1}{n}
            \mE_{n,12}^\trans \mE_{n,12}.
\end{align}
\end{subequations}

Now we study the eigendecomposition of $\mS_n$.  If 
\(
    \vv
    = 
    \left( 
    \begin{matrix}
        \vv_{1} \\
        \vv_{2}
    \end{matrix}
    \right)
\)
is an eigenvector of $\mS_n$ with eigenvalue $\mu$, then
\[
    \left(
    \begin{matrix}
        \mS_{n,11} & \mS_{n,12} \\
        \mS_{n,21} & \mS_{n,22}
    \end{matrix}
    \right)
    \left( 
    \begin{matrix}
        \vv_{1} \\
        \vv_{2}
    \end{matrix}
    \right)
    =
    \mu
    \left( 
    \begin{matrix}
        \vv_{1} \\
        \vv_{2}
    \end{matrix}
    \right).
\]
If $\mu$ is not an eigenvalue of $\mS_{n,22}$, then we get
\begin{subequations}
\begin{gather}\label{E:v2-from-v1}
    \vv_2
    =
    -
    \left(
        \mS_{n,22}
        -
        \mu
        \mI_{p-k}
    \right)^{-1}
    \mS_{n,21} \,
    \vv_1, \qquad\text{and} \\
    \label{E:secular-nof}
    \left(
        \mS_{n,11}
        -
        \mu
        \mI_k
        -
        \mS_{n,12}
        \left(
            \mS_{n,22}
            -
            \mu
            \mI_{p-k}
        \right)^{-1}
        \mS_{n,21}
    \right)
    \vv_1
    =
    0.
\end{gather}
\end{subequations}
Conversely, if $(\mu, \vv_1)$ is a pair that solves~\eqref{E:secular-nof}
and $\vv_1 \neq 0$, then
\begin{equation}\label{E:v-from-v1}
    \vv
    =
    \left(
    \begin{matrix}
        \vv_1 \\
        -
        \left(
            \mS_{n,22}
            -
            \mu
            \mI_{p-k}
        \right)^{-1}
        \mS_{n,21} \,
        \vv_1
    \end{matrix}
    \right)
\end{equation}
is an eigenvector of $\mS_n$ with eigenvalue $\mu$.  

We define
\begin{gather}
    \mT_n(z)
        =
            \mS_{n,11}
            -
            z
            \mI_k
            -
            \mS_{n,12}
            \left(
                \mS_{n,22}
                -
                z
                \mI_{p-k}
            \right)^{-1}
            \mS_{n,21}
            \label{E:secular-T} \\
    f_n ( z, \vx )
        =
            \mT_n(z) \,
            \vx, \label{E:secular-f}
\end{gather}
and refer to $f_n(z, \vx) = 0$ as the \emph{secular equation}. 
This terminology comes from numerical linear algebra, where a secular equation is analogous to a characteristic equation; it is an equation whose
roots are eigenvalues of a matrix.   Typically, secular equations arise in eigenvalue perturbation problems.  The name comes from the fact that they originally appeared studying secular perturbations of planetary orbits.  A more standard use of the term ``secular equation'' would involve the equation $\det \mT_n(z) = 0$.  However, for our purposes it is more convenient to work with $f_n$.

\subsection{Outline of the proof}

Almost surely $\mS_n$ and $\mS_{n,22}$ have no eigenvalues in common, so every eigenvalue-eigenvector pair of $\mS_n$ is a solution to the secular equation.  To study these solutions, we first focus on $\mT_n(z)$.  

It turns out that when $z > (1 + \gamma^{-1/2})^2$, we can find a perturbation expansion for $\mT_n(z)$. In Section~\ref{S:anal-sec-eqn}, we show that for $z$ above this threshold, we can expand
\[
    \mT_n(z)
        =
            \mT_0(z)
            +
            \frac{1}{\sqrt{n}}
            \mT_{n,1}(z),
\]
where $\mT_0(z)$ is deterministic and $\mT_{n,1}(z)$ converges in distribution
to a matrix-valued Gaussian process indexed by $z$.  With this expansion, in Section~\ref{S:solutions-to-secular-eq} we study sequences of solutions to the equation $f_n(z_n, \vx_n) = 0$.  Using a Taylor series expansion for $z_n > (1 + \gamma^{-1/2})^2$, we write
\begin{align*}
    z_n
        &= 
            z_0 
            + 
            \frac{1}{\sqrt{n}}
            z_{n,1} 
            + 
            \ohP\left( \frac{1}{\sqrt{n}} \right), \\
    \vx_n
        &=
            \vx_0
            +
            \frac{1}{\sqrt{n}}
            \vx_{n,1}
            + 
            \ohP\left( \frac{1}{\sqrt{n}} \right),
\end{align*}
where $(z^0, \vx^0)$ is the limit of $(z_n, \vx_n)$ as $n\to \infty$ and
$(z_{n,1}, \vx_{n,1})$ is the order-$\frac{1}{\sqrt{n}}$ approximation error.

In Section~\ref{S:singular-values-and-vectors} we get the singular values and singular vectors of $\frac{1}{\sqrt{n}} \mtX_n$.  From every solution pair $(z_n, \vx_n)$ satisfying $f_n(z_n, \vx_n) = 0$, we can construct an eigenvalue and an eigenvector of $\mS_n$ using equation~\eqref{E:v-from-v1}.
The eigenvalues are squares of singular values of $\frac{1}{\sqrt{n}}\mtX_n$, and the eigenvectors are right singular vectors.  We can get the corresponding left singular vectors by multiplying the right singular vectors by $\mtX_n$ and scaling appropriately.  For $z$ values above the critical threshold, we use the perturbation results of the previous two sections.  Below the threshold, we use a more direct approach involving the fluctuations of the top eigenvalues of $\mLambda_n$.

Finally, in Section~\ref{S:lowrank-gamma-lt-1} we show that the results still hold when $\gamma < 1$.  For parts of the proof, we will need some limit theorems for weighted sums from 
Appendix~\ref{A:weighted-sums}.

\section{Analysis of the secular equation}\label{S:anal-sec-eqn}

We devote this section to finding a simplified formula for $\mT_n(z)$ for certain values of $z$.  By a bit of algebra and analysis, we find the first- and second-order behavior of the secular equation.

First, we employ the Sherman-Morrison-Woodbury formula~\cite{golub1996mc} to  get an expression for $\left(\mS_{n,22} - z \mI_{p-k}\right)^{-1}$.  As a reminder, this identity states that for matrices $\mA$, $\mB$, and $\mC$ with compatible dimensions, the following formula for the inverse holds:
\[
    \left(
        \mA
        +
        \mB
        \mC
    \right)^{-1}
        =
        \mA^{-1}
        -
        \mA^{-1}
        \mB
        \left(
            \mI
            +
            \mC
            \mA^{-1}
            \mB
        \right)^{-1}
        \mC
        \mA^{-1}.
\]
Using this, we can write
\begin{align}\label{E:lowrank-smw-expansion}
    \left(
        \mS_{n,22} - z \mI_{p-k}
    \right)^{-1}
        &=
            \left(
                \left(
                    \mLambda_n - z \mI_{p-k}
                \right)
                +
                \frac{1}{n}
                \mE_{n,12}^\trans \mE_{n12}
            \right)^{-1} \notag \\
        \begin{split}
        &=
            \left(
                \mLambda_n
                -
                z
                \mI_{p-k}
            \right)^{-1} \\
            &\qquad-
            \frac{1}{n}
            \left(
                \mLambda_n
                -
                z
                \mI_{p-k}
            \right)^{-1} \\
            &\qquad\qquad\cdot
            \mE_{n,12}^\trans
            \left(
                \mI_k
                +
                \frac{1}{n}
                \mE_{n,12}
                \left(
                    \mLambda_n
                    -
                    z
                    \mI_{p-k}
                \right)^{-1}
                \mE_{n,12}^\trans
            \right)^{-1}
            \mE_{n,12} \\
            &\qquad\qquad\cdot
            \left(
                \mLambda_n
                -
                z
                \mI_{p-k}
            \right)^{-1}.
    \end{split}
\end{align}
Next, we define $\mtD_n = \mD_n + \frac{1}{\sqrt{n}} \mE_{n,11}$, so that
\begin{align}\label{E:lowrank-S-schur}
    \begin{split}
    \mS_{n,12} 
    \Big(
        \mS_{n,22} &- z \mI_{p-k}
    \Big)^{-1}
    \mS_{n,21} \\
        &=
            \frac{1}{n}
            \left(
                \mtD_n^\trans \mE_{n,12}
                +
                \mE_{n,21}^\trans \mLambda_n^{1/2}
            \right) \\
            &\qquad\cdot
            \left(
                \mS_{n,22} - z \mI_{p-k}
            \right)^{-1} \\
            &\qquad\cdot
            \left(
                \mE_{n,12}^\trans \mtD_n
                +
                \mLambda_n^{1/2} \mE_{n,21}
            \right).
    \end{split}
\end{align}

There are three important terms coming out of equations~\eqref{E:lowrank-smw-expansion}~and~\eqref{E:lowrank-S-schur} that involve $\mLambda_n$.  These are
\(
    \mE_{n,12}
    \left(
        \mLambda_n - z \mI_{p-k}
    \right)^{-1}
    \mE_{n,12}^\trans,
\)
\(
    \mE_{n,21}^\trans
    \left(
        \mLambda_n - z \mI_{p-k}
    \right)^{-1}
    \mE_{n,21},
\)
and
\(
    \mE_{n,12}
    \left(
        \mLambda_n - z \mI_{p-k}
    \right)^{-1}
    \mLambda_n^{1/2}
    \mE_{n,21}.
\)
Each term can be written as a weighted sum of outer products.  

There is
dependence between the weights, but the outer products are iid.  For example,
with
\(
    \mE_{n,12}
    =
    \left(
    \begin{matrix}
        \vE_{n,12,1} &
        \vE_{n,12,2} &
        \cdots &
        \vE_{n,12,p-k}
    \end{matrix}
    \right),
\) 
we have
\[
    \mE_{n,12}
    \left(
        \mLambda_n - z \mI_{p-k}
    \right)^{-1}
    \mE_{n,12}^\trans,
        =
            \sum_{\alpha=1}^{p-k}
                \frac{1}{\lambda_{n,\alpha} - z}
                \cdot
                \vE_{n,12,\alpha} \,
                \vE_{n,12,\alpha}^\trans.
\]
From the Central Limit Theorem and the Strong Law of Large Numbers
we know that
\(
    \frac{1}{p-k}
    \sum_{\alpha=1}^{p-k}
        \vE_{n,12,\alpha} \,
        \vE_{n,12,\alpha}^\trans
        \toas
            \mI_{k}
\)
and that
\(
    \sqrt{p-k}
    \left(
        \frac{1}{p-k}
        \sum_{\alpha=1}^{p-k}
            \vE_{n,12,\alpha} \,
            \vE_{n,12,\alpha}^\trans
        -
        \mI_{k}
    \right)
\)
converges in distribution to mean-zero symmetric matrix whose elements are jointly multivariate normal.  In the limiting distribution, the elements have variance $2$ along the diagonal and variance $1$ off of it; aside from the matrix being symmetric, the unique elements are all uncorrelated.  The
difficulty in analyzing these terms comes from the dependence between 
the weights.

When $z$ is in the support of $\FMP_\gamma$, the weights behave
erratically, but otherwise they have some nice properties.
First of all, $\mLambda_n$ is independent of $\mE_{n,12}$ and $\mE_{n,21}$.
Secondly, the Wishart LLN~(Corollary~\ref{C:wishart-lln}) and
Theorem~\ref{T:max-wishart-eig-limit} ensure that for
$z~>~(1+\gamma^{-1/2})^2$,
\(
    \frac{1}{p-k}
    \sum_{\alpha=1}^{p-k}
        \frac{1}{\lambda_{n,\alpha} - z}
    \toas
    \int
        \frac{1}{t - z}
        d\FMP_\gamma(t).
\)
Moreover, the Wishart CLT~(Theorem~\ref{T:wishart-clt}) guarantees that
the error is of size $\OhP(\frac{1}{n})$.  These properties in
combination with the limit theorems for weighted sums in 
Appendix~\ref{A:weighted-sums} allow us to get the behavior of
\(
    \mE_{n,12}
    \left(
        \mLambda_n
        -
        z
        \mI_k
    \right)^{-1}
    \mE_{n,12}^\trans
\)
and its cousins.

The function
\begin{align}
    m(z)
        &\define
            \int
                \frac{1}{t - z} d\FMP_\gamma(t) \notag \\
        &=
            \gamma
            \cdot
            \frac{ -(z - 1 + \gamma^{-1})
                   +\sqrt{ \big( z - b_\gamma \big) 
                           \big( z - a_\gamma \big) } }
                 { 2 z }
\end{align}
is the Stieltjes transform of $\FMP_\gamma$, 
where
\(
    a_\gamma = \left( 1 - \gamma^{-1/2} \right)^2
\)
and
\(
    b_\gamma = \left( 1 + \gamma^{-1/2} \right)^2.
\)
When restricted to the complement of the support of $\FMP_\gamma$, $m$ has a well-defined inverse
\begin{equation}\label{E:stieltjes-inverse}
    z(m)
        = 
            -
            \frac{1}{m}
            +
            \frac{1}{1 + \gamma^{-1} \, m}.
\end{equation}
Also, $m$ is strictly increasing and convex outside the support of
$\FMP_\gamma$.  This function appears frequently in the remainder of the chapter.

\begin{lemma}\label{L:eij-product-limits}
    If $z > b_\gamma$, then
    \begin{subequations}
    \begin{align}
        \frac{1}{n}
        \mE_{n,12}
        \left(
            \mLambda_n
            -
            z \mI_{p-k}
        \right)^{-1}
        \mE_{n,12}^\trans
            &\toas
                \gamma^{-1}
                m(z)
                \cdot
                \mI_k, \\
        \frac{1}{n}
        \mE_{n,21}^\trans
        \left(
            \mLambda_n
            -
            z \mI_{p-k}
        \right)^{-1}
        \mE_{n,21}
            &\toas
                \gamma^{-1}
                m(z)
                \cdot
                \mI_k,
    \end{align}
    and
    \begin{equation}
        \frac{1}{n}
        \mE_{n,12}
        \left(
            \mLambda_n
            -
            z \mI_{p-k}
        \right)^{-1}
        \mLambda_n^{1/2}
        \mE_{n,21}
            \toas
                0.
    \end{equation}
    \end{subequations}
\end{lemma}
\begin{proof}
We prove the result for the first quantity and the other derivations are analogous.  For each $1 \leq i, j \leq k$, we have that
\begin{align*}
    \left(
        \frac{1}{n}
        \mE_{n,12}
        \left(
            \mLambda_n
            -
            z \mI_{p-k}
        \right)^{-1}
        \mE_{n,12}^\trans    
    \right)_{ij}
    =
    \frac{p-k}{n}
    \cdot
    \frac{1}{p-k}
    \sum_{\alpha=1}^{p-k}
        \frac{ \left( \mE_{n,12} \right)_{i\alpha}
               \left( \mE_{n,12} \right)_{j\alpha}
             }
             { \lambda_{n,\alpha} - z }.
\end{align*}
Let $N = p-k$, define weights 
\(
    W_{N,\alpha} = (\lambda_{n,\alpha} - z)^{-1},
\)
and let
\(
    Y_{N,\alpha} = \left( \mE_{n,12} \right)_{i\alpha}
                   \left( \mE_{n,12} \right)_{j\alpha}.
\)
The function
\[
    g(t) 
    = 
    \begin{cases}
        \frac{1}{t - z} 
            &\text{if $t \leq b_\gamma$,} \\
        \frac{1}{b_\gamma - z}
            &\text{otherwise,}
    \end{cases}
\]
is bounded and continuous.  Moreover, since $\lambda_{n,1} \toas b_\gamma$, with probability one $g(\lambda_{n,\alpha})$ is eventually equal to $W_{N,\alpha}$ for all $\alpha$.  The Wishart LLN (Corollary~\ref{C:wishart-lln}) gives us that
\(
    \frac{1}{N}
    \sum_{\alpha=1}^N W_{N,\alpha}
        \toas
            \int
                \frac{1}{t - z} d\FMP_\gamma(t) = m(z).
\)
Since $| W_{N,\alpha} | \leq W_{N,1} \overset{\text{a.s.}}{\leq} b_\gamma$, the fourth moments of the weights are uniformly bounded in $N$.  
The $Y_{N,\alpha}$ are all iid with $\E Y_{N,\alpha} = \delta_{ij}$ and $\E Y_{N,\alpha}^4 < \infty$.  
Applying these results, the weighed SLLN (Proposition~\ref{P:weighted-slln}) gives us that
\(
    \frac{1}{N}
    \sum_{\alpha=1}^N
        W_{N,\alpha}
        Y_{N,\alpha}
    \toas
        m(z)
        \delta_{ij}.
\)
Since $\frac{p-k}{n} \to \gamma^{-1}$, this completes the proof.
\end{proof}


\begin{lemma}\label{L:eij-product-limits-uniform}
    Considered as functions of $z$, the quantities in 
    Lemma~\ref{L:eij-product-limits} and their derivatives
    converge uniformly over any closed
    interval $[u,v] \subset \big( b_\gamma,\infty \big)$.
\end{lemma}
\begin{proof}
    We show this for the first quantity and the other proofs are similar.
    Defining
    \[
        e_{n,ij}(z)
        =
        \left(
            \frac{1}{n}
            \mE_{n,12}
            \left(
                \mLambda_n - z \mI_{p-k}
            \right)^{-1}
            \mE_{n,12}^\trans
        \right)_{ij},
    \]
    we will show that for all $(i,j)$ with $1 \leq i,j \leq k$, 
    \(
        \sup_{z \in [u,v]} 
            \left|
                e_{n,ij}(z)
                    -
                    \gamma^{-1}
                    m(z)
                    \delta_{ij}
            \right|
            \toas 0.
    \)
    
    Let $\epsilon > 0$ be arbitrary.  Note that for 
    $z > b_\gamma$, $m'(z) > 0$ and $m''(z) < 0$.  We let $d = m'(u)$ and
    choose a grid
    \(
        u = z_1 < z_2 < \cdots < z_{M-1} < z_M = v,
    \)
    with $|z_l - z_{l+1}| = \frac{\gamma \epsilon}{2 d}$.  
    Then $|\gamma^{-1} m(z_l) - \gamma^{-1} m(z_{l+1})| < \frac{\epsilon}{2}$.  
    From 
    Lemma~\ref{L:eij-product-limits}, we can find $N$ large enough such
    that for $n > N$, 
    \(
        \max_{l\in\{1, \ldots, M\}}
            \left|
                e_{n,ij}(z_l)
                    -
                    \gamma^{-1}
                    m(z_l)
                    \delta_{ij}
            \right|
            <
            \frac{\epsilon}{2}.
    \)
    Also guarantee that $N$ is large enough so that $\lambda_{n,1} < u$ 
    (this is possible since $\lambda_{n,1} \toas b_\gamma < u$).
    Let $z \in [u,v]$ be arbitrary and find $l$ such that 
    $z_l \leq z \leq z_{l+1}$.
    Observe that $e_{n,ij}(z)$ is monotone for $z > \lambda_{n,1}$.  Thus,
    either
    \(
        e_{n,ij}(z_l) \leq e_{n,ij}(z) \leq e_{n,ij}(z_{l+1})
    \)
    or
    \(
        e_{n,ij}(z_{l+1}) \leq e_{n,ij}(z) \leq e_{n,ij}(z_{l}).
    \)
    
    If $i \neq j$, we have for $n > N$,
    \(
        -\frac{\epsilon}{2} < e_{n,ij}(z) < \frac{\epsilon}{2}.
    \)
    Otherwise, when $i = j$, $e_{n,ij}(z)$ is 
    monotonically increasing and
    \(
        \gamma^{-1} m(z_l) - \frac{\epsilon}{2}
            <
                e_{n,ij}(z)
                    <
                        \gamma^{-1} m(z_{l+1}) + \frac{\epsilon}{2},
    \)
    so that
    \(
        \gamma^{-1} m(z) - \epsilon
            <
                e_{n,ij}(z)
                    <
                        \gamma^{-1} m(z) + \epsilon.
    \)
    In either case, 
    $|e_{n,ij}(z) - \gamma^{-1} m(z) \delta_{ij}| < \epsilon$.
    Since 
    \(
        \frac{d}{dz} \left[
            \frac{1}{\lambda - z}
        \right]
            =
            -\frac{1}{(\lambda - z)^{2}},
    \)
    which is monotone for $z > \lambda$, the same argument applies to show
    that the derivatives converge uniformly.
\end{proof}


\begin{lemma}\label{L:eij-product-scaled-limits}
    If $z_1, z_2, \ldots, z_l > b_\gamma$, 
    then jointly for $z \in \{ z_1, z_2, \ldots, z_l \}$
    \begin{subequations}
    \begin{align}
        \sqrt{n}
        \left(
            \frac{1}{n}
            \mE_{n,12}
            \left(
                \mLambda_n
                -
                z
                \mI_{p-k}
            \right)^{-1}
            \mE_{n,12}^\trans
            -
            \gamma^{-1}
            m(z)
            \mI_k
        \right) 
            &\define \mF_n(z)
            \tod \mF(z), \\
        \sqrt{n}
        \left(
            \frac{1}{n}
            \mE_{n,12}
            \left(
                \mLambda_n
                -
                z
                \mI_{p-k}
            \right)^{-1}
            \mLambda_{n}^{1/2}
            \mE_{n,21}
        \right) 
            &\define \mG_n(z)
            \tod \mG(z), \\
        \sqrt{n}
        \left(
            \frac{1}{n}
            \mE_{n,21}^\trans
            \left(
                \mLambda_n
                -
                z
                \mI_{p-k}
            \right)^{-1}
            \mE_{n,21}
            -
            \gamma^{-1}
            m(z)
            \mI_k
        \right) 
            &\define \mH_n(z)
            \tod \mH(z),
    \end{align}
    \end{subequations}
    where the elements of $\mF(z)$, $\mG(z)$, and $\mH(z)$
    jointly define a multivariate Gaussian process indexed by $z$,
    with each matrix independent of the others. 
    The nonzero covariances are defined by
    \begin{subequations}
    \begin{align}
        \cov\Big( F_{ij}(z_1), \, F_{ij}(z_2) \Big)
            &= \gamma^{-1}
               \left( 1 + \delta_{ij} \right)
               \frac{ m(z_1) - m(z_2) }
                    { z_1 - z_2 }, \\
        \cov\Big( G_{ij}(z_1), \, G_{ij}(z_2) \Big)
            &= \gamma^{-1} \,
               \frac{ z_1 \, m(z_1) - z_2 \, m(z_2) }
                    { z_1 - z_2 }, \\
        \cov\Big( H_{ij}(z_1), \, H_{ij}(z_2) \Big)
            &= \gamma^{-1}
               \left( 1 + \delta_{ij} \right)
               \frac{ m(z_1) - m(z_2) }
                    { z_1 - z_2 },
    \end{align}
    \end{subequations}
    with the interpretation when $z_1 = z_2$ that
    \[
        \frac{ m(z_1) - m(z_2) }
             { z_1 - z_2 }
            =
                m'(z_1),
        \quad\text{and}\quad
        \frac{ z_1 \, m(z_1) - z_2 \, m(z_2) }
             { z_1 - z_2 }
            = m( z_1 ) + z_1 \, m'(z_1).
    \]
\end{lemma}
\begin{proof}
    We will need the Wishart CLT~(Theorem~\ref{T:wishart-clt}) and the strong 
    weighted multivariate CLT~(Corollary~\ref{C:strong-weighted-clt}).  To 
    save space we only
    give the argument for the joint distribution of $\mF(z_1)$ and 
    $\mF(z_2)$.
    
    Put $N = p-k$ and consider the $2 k^2$-dimensional vector
    \(
        \vY_{N,\alpha}
        =
        \left(
        \begin{matrix}
            \vtY_{N,\alpha} \\
            \vtY_{N,\alpha}
        \end{matrix}
        \right),
    \)
    where
    \(
        \vtY_{N,\alpha}
        =
        \vecm \left(
            \vE_{n,12,\alpha} \, \vE_{n,12,\alpha}^\trans
        \right)
    \)
    and
    \(
        \mE_{n,12}
        =
        \left(
        \begin{matrix}
            \vE_{n,12,1} &
            \vE_{n,12,2} &
            \cdots &
            \vE_{n,12,N} &            
        \end{matrix}
        \right).
    \)
    Define the $2 k^2$-dimensional weight vector
    \(
        \vW_{N,\alpha}
        =
        \left(
        \begin{matrix}
            \vW_{N,\alpha,1} \\
            \vW_{N,\alpha,2}
        \end{matrix}
        \right),
    \)
    where
    \(
        \vW_{N,\alpha,i}
        =
        \frac{1}{\lambda_\alpha - z_i}
        \vone.
    \)
    We have that for $\alpha = 1, 2, \ldots, N$, the $\vY_{N,\alpha}$ are
    iid with
    \[
        \E \left[ \vY_{N,1} \right]
        =
        \vmu^Y
        =
        \left(
        \begin{matrix}
            \vtmu^Y \\
            \vtmu^Y
        \end{matrix}
        \right)
    \]
    and $\vtmu^Y = \vecm\left( \mI_k \right)$.  Also, we have
    \[
        \E \left[ 
            \left(
                \vY_{N,1}
                -
                \vmu^Y
            \right)
            \left(
                \vY_{N,1}
                -
                \vmu^Y
            \right)^\trans
        \right]
        =
        \mSigma^Y
        =
        \left(
        \begin{matrix}
            \mtSigma^Y & \mtSigma^Y \\
            \mtSigma^Y & \mtSigma^Y
        \end{matrix}
        \right),
    \]
    where
    \(
        \mtSigma^Y
        =
        \E \Big[
            \Big( \!
                \vecm\left( 
                    \vE_{n,12,1} \vE_{n,12,1}^\trans - \mI_k
                \right) \! \!
            \Big)
            \Big( \!
                \vecm\left( 
                    \vE_{n,12,1} \vE_{n,12,1}^\trans - \mI_k
                \right) \! \!
            \Big)^{\! \trans}
        \Big]
    \)
    is a $k^2 \times k^2$ matrix defined by the relation
    \[
        \E \left[
            \left( 
                \vE_{n,12,1} \vE_{n,12,1}^\trans - \mI_k
            \right)_{ij}
            \left( 
                \vE_{n,12,1} \vE_{n,12,1}^\trans - \mI_k
            \right)_{i'j'}
        \right]
        =
        \delta_{(i,j) = (i',j')}
        +
        \delta_{(i,j) = (j',i')}.
    \]
    That is, the diagonal elements of 
    $\vE_{n,12,1} \vE_{n,12,1}^\trans - \mI_k$ have variance $2$ and
    the off-diagonal elements have variance $1$.  Aside from the matrix
    being symmetric, the unique elements are all uncorrelated.
    
    Letting
    \begin{align*}
        \mu^W_i
            &=
                \int \frac{ d\FMP_\gamma(t) }
                          { t - z_i }
            =
                m( z_i ), \qquad\text{and} \\
        \sigma^W_{ij}
            &=
                \int \frac{ d\FMP_\gamma(t) }
                          { (t - z_i)(t - z_j) }
            =
                \frac{ m( z_i ) - m( z_j ) }
                     { z_i - z_j },
    \end{align*}
    the Wishart LLN combined with the truncation argument of
    Lemma~\ref{L:eij-product-limits} gives us that
    \(
        \frac{1}{N}
        \sum_{\alpha=1}^N
            \frac{1}{  \lambda_{n,\alpha} - z_i  }
            \toas
                \mu^W_{i}
    \)
    and
    \(
        \frac{1}{N}
        \sum_{\alpha=1}^N
            \frac{1}{ ( \lambda_{n,\alpha} - z_i )
                      ( \lambda_{n,\alpha} - z_j ) }
            \toas
                \sigma^W_{ij}.
    \)
    Thus,
    \begin{align*}
        \frac{1}{N}
        \sum_{\alpha=1}^N
            \vW_{N,\alpha}
            &\toas
                \vmu^W =
                \left(
                \begin{matrix}
                    \mu^W_1 \vone \\
                    \mu^W_2 \vone
                \end{matrix}
                \right), \qquad\text{and} \\
        \frac{1}{N}
        \sum_{\alpha=1}^N
            \vW_{N,\alpha} \,
            \vW_{N,\alpha}^\trans
            &\toas
                \mSigma^W
            =
                \left(
                \begin{matrix}
                    \sigma^W_{11} \vone \, \vone^\trans &
                        \sigma^W_{12} \vone \, \vone^\trans \\
                    \sigma^W_{21} \vone \, \vone^\trans &
                        \sigma^W_{11} \vone \, \vone^\trans
                \end{matrix}
                \right).
    \end{align*}
    Moreover, the Wishart CLT tells us that the error in each of the
    sums is of size $\OhP\left( \frac{1}{N} \right)$.  
        
    As in Lemma~\ref{L:eij-product-limits}, the fourth moments of
    $\vW_{N,\alpha}$ and $\vY_{N,\alpha}$ are all well-behaved.  Finally,
    we can invoke the strong weighted CLT 
    (Corollary~\ref{C:strong-weighted-clt}) to get that the weighted sum
    \(
        \sqrt{N} \left(
            \frac{1}{N}
            \sum_{\alpha=1}^N
                \vW_{N,\alpha} \bullet \vY_{N,\alpha}
            -
                \vmu^W \bullet \vmu^T
        \right)
    \)
    converges in distribution to a mean-zero multivariate normal with
    covariance
    \[
        \mSigma^W \bullet \mSigma^Y
            =
                \left(
                \begin{matrix}
                    \sigma^W_{11} \mtSigma &
                        \sigma^W_{12} \mtSigma \\
                    \sigma^W_{21} \mtSigma &
                        \sigma^W_{22} \mtSigma 
                \end{matrix}
                \right).
    \]
    This completes the proof since
    \begin{multline*}
        \sqrt{n}
        \vecm \left(
            \frac{1}{n}
            \mE_{n,12}
            \left(
                \mLambda_n
                -
                z_i
                \mI_{p-k}
            \right)^{-1}
            \mE_{n,12}^\trans
            -
            \gamma^{-1}
            m( \mu )
            \mI_k
        \right) \\
            =
                \gamma^{-1}
                \sqrt{ \frac{n}{p-k} }
                \cdot
                \sqrt{ N }
                \Bigg( 
                    \frac{1}{N}
                    \sum_{\alpha=1}^N
                        \vW_{N,\alpha,i} \bullet \vtY_{N,\alpha}
                    -
                        \vmu_i^W \bullet \vtmu_i^Y
                \Bigg)
    \end{multline*}
    (with $\vmu_i^W = \mu_i^W \vone$),
    and
    \(
        \gamma^{-1}
        \sqrt{ \frac{n}{p-k} }
        \to
        \gamma^{-1/2}.
    \)
\end{proof}

\begin{remark}
    It is possible to show that the sequences in Lemma~\ref{L:eij-product-scaled-limits} are tight by an argument similar to the one used in \cite{onatski2009}.  This implies that the convergence is uniform in $z$.  For our purposes, we only need that the finite-dimensional distributions converge.
\end{remark}

With Lemmas~\ref{L:eij-product-limits}--\ref{L:eij-product-scaled-limits},
we can get a perturbation expansion of $\mT_{n}(z)$ for $z > b_\gamma$.

\begin{lemma}
    If $[u,v] \subset \big( b_\gamma, \, \infty \big)$, then
    \(
        \mT_n( z )
            \toas
                \mT_0( z )
    \)
    uniformly on $[u,v]$, where
    \begin{equation}
        \mT_0( z )
            =
                \frac{1}{ 1 + \gamma^{-1} m(z) }
                \mD^2 
                +
                \frac{1}{m(z)}
                \mI_k.
    \end{equation}
\end{lemma}

\begin{lemma}
    If $z > b_\gamma$, then
    \begin{equation}
        \mT_n( z ) 
            = 
                \mT_0( z )
                +
                \frac{1}{\sqrt{n}}
                \mT_{n,1}( z ),
    \end{equation}
    where
    \begin{align}
            \mT_{n,1}( z )
                = 
                &-
                \big( 1 + \gamma^{-1} m(z) \big)^{-2}
                    \cdot
                    \mD \mF_n(z) \mD \notag \\
                &+
                \big( 1 + \gamma^{-1} m(z) \big)^{-1} \notag \\
                    &\quad\quad\quad\cdot
                    \Big\{
                        \sqrt{n} \left( \mD_n^2 - \mD^2 \right)
                        + 
                        \mD \big( \mE_{n,11} - \mG_n(z) \big)
                        + 
                        \big( \mE_{n,11} - \mG_n(z) \big)^\trans \mD
                    \Big\}  \notag \\
                &-
                z \, \mH_n(z)
                +
                \sqrt{n}
                \left(
                    \frac{1}{n} \mE_{n,31}^\trans \mE_{n,31}
                    -
                    (1 - \gamma^{-1})
                    \mI_k
                \right)
                +
                \ohP\left( 1 \right).
    \end{align}
\end{lemma}
\begin{proof}
    First, we have
    \begin{multline*}
        \mS_{n,11}
            = 
                \mD_n^2 + \mI_k
                +
                \frac{1}{\sqrt{n}}
                \left(
                    \mD \mE_{n,11}
                    +
                    \mE_{n,11}^\trans \mD
                \right) \\
                +
                \left(
                    \frac{1}{n}
                    \big(
                        \mE_{n,11}^\trans \mE_{n,11}
                        +
                        \mE_{n,21}^\trans \mE_{n,21}
                        +
                        \mE_{n,31}^\trans \mE_{n,31}
                    \big)
                    -
                    \mI_k
                \right).
    \end{multline*}
    Next, we compute
    \begin{align*}
        \begin{split}
        \Bigg(
            \mI_k
            +
            \frac{1}{n}
            \mE_{n,12}
            &\Big(
                \mLambda_n - z \mI_{p-k}
            \Big)^{-1}
            \mE_{n,12}^\trans
        \Bigg)^{-1} \\
            &=
                \left(
                    \mI_k
                    +
                    \gamma^{-1}
                    m( z )
                    \mI_k
                    +
                    \frac{1}{\sqrt{n}}
                    \mF_n(z)
                \right)^{-1} 
        \end{split} \\
%            &=
%                \big(1 + \gamma^{-1} m(z)\big)^{-1}
%                \left(
%                    \mI_k
%                    +
%                    \frac{1}{\sqrt{n}}
%                    \big( 1 + \gamma^{-1} m(z) \big)^{-1}
%                    \mF_{n}(z)
%                \right)^{-1} \\
            &=
                \big( 1 + \gamma^{-1} m(z) \big)^{-1} \mI_k
                -
                \frac{1}{\sqrt{n}}
                \big( 1 + \gamma^{-1} m(z) \big)^{-2}
                \mF_{n}(z)
                +
                \ohP\left( \frac{1}{\sqrt{n}} \right).              
    \end{align*}
    Using this, after a substantial amount of algebra we get
    \[
        \begin{split}
            \mS_{n,12} 
            &\Big( \mS_{n,22} - z \mI_{p-k} \Big)^{-1} 
            \mS_{n,21} \\
                &=
                z \, m(z) \, \mI_k
                +
                \frac{\gamma^{-1} m(z)}
                     { 1 + \gamma^{-1} m(z)} \mD_n^2 \\
                &\quad+
                \frac{1}{\sqrt{n}} \Bigg\{
                    \frac{1}{1 + \gamma^{-1} m(z)}
                        \left( 
                            \mD \mG_n(z) 
                            + \mG_n^\trans(z) \mD 
                        \right) \\
                    &\quad\qquad\qquad+
                    \frac{\gamma^{-1} m(z)}{1 + \gamma^{-1} m(z)}
                    \left( \mD \mE_{n,11} + \mE_{n,11}^\trans \mD \right) \\
                    &\quad\qquad\qquad+
                    \left(
                        \frac{1}{1 + \gamma^{-1} m(z)}
                    \right)^2
                    \mD \mF_n(z) \mD
                    +
                    z \, \mH_n( z ) \Bigg\} \\
                &\quad+
                \frac{1}{n} \mE_{n,21}^\trans \mE_{n,21}
                +
                \ohP\left( \frac{1}{\sqrt{n}} \right).
        \end{split}
    \]
    The equations for $\mT_0$ and $\mT_{n,1}$ follow.  
    To simplify the form of $\mT_0$, we use the identity
    \(
        z
        \cdot
        \Big(
            1
            +
            \gamma^{-1} m(z)
        \Big)
        =
        -
        \frac{1}{m(z)}
        +
        (1 - \gamma^{-1}).
    \)
\end{proof}


\section{Solutions to the secular equation}\label{S:solutions-to-secular-eq}

We will now study the solutions to $f_{n}(z, \vx) = 0$, defined in 
equation~\eqref{E:secular-f}.  If $(z, \vx)$
is a solution, then so is $(z, \alpha \, \vx)$ for any scalar $\alpha$.  We restrict our attention to solutions with $\|\vx \|_2 = 1$.  We also impose a restriction on the sign of $\vx$, namely we require that the component with the largest magnitude is positive, i.e.
\begin{equation}\label{E:x-identifiability}
    \max_i x_i = \max_i | x_i |.
\end{equation}

\subsection{Almost sure limits}

First we look at the solutions of $f_0(z, \vx) \define \mT_0(z) \vx$, the limit of $f_n(z, \vx)$ for $z > b_\gamma$.  

\begin{lemma}
    Letting $\bar k = \max \{ i : \mu_i > \gamma^{-1/2} \}$, if $\mu_1, \mu_2, 
    \ldots, \mu_k$ are all distinct, then there are exactly
    $\bar k$ solutions to the equation $f_0( z, \vx ) = 0$.  They are given
    by
    \[
        ( \bmu_i, \ve_i ), \quad \text{$i = 1, \ldots, \bar k$},
    \]
    where $\bmu_i$ is the unique solution
    \[
        m( \bmu_i ) = \frac{1}{\mu_i + \gamma^{-1}},
    \]
    and $\ve_i$ is the $i$th standard basis vector.
\end{lemma}
\begin{proof}
    We have that 
    \[
        \mT_0(z) 
        = 
        \frac{1}{1 + \gamma^{-1} m(z)} \mD^2 
        + 
        \frac{1}{m(z)} \mI_k.
    \]
    Since this is diagonal, the equation $f_0(z, \vx) = 0$ holds iff the 
    $i$th diagonal element of $\mT_0(z)$ is zero and $\vx = \ve_i$.  The 
    $i$th diagonal is zero when 
    \[
        \frac{\mu_i}{1 + \gamma^{-1} m(z)} + \frac{1}{m(z)} = 0,
    \]
    equivalently
    \[
        m(z) = - \frac{1}{\mu_i + \gamma^{-1}}.
    \]
    Note that $m(z) > - \left( \gamma^{-1/2} + \gamma^{-1} \right)^{-1}$ and
    $m'(z) > 0$ on $\big( b_\gamma, \infty \big)$.  Hence, a unique solution
    exists exactly when $\mu_i > \gamma^{-1/2}$.
\end{proof}

Given a solution of $f_0(z, \vx) = 0$, it is not hard to believe that there is a sequence of solutions $(z_n, \vx_n)$ such that $f_n(z_n, \vx_n) = 0$, with $z_n$ and $\vx_n$ converging to $z$ and $\vx$, respectively.  We dedicate the rest of this section to making this statement precise.  

\begin{lemma}\label{L:zn-sequence-exists}
    If $\mu > \gamma^{-1/2}$ occurs $l$ times on the diagonal of $\mD^2$, then
    with probability one there exist sequences $z_{n,1}, 
    \ldots, z_{n,l}$ such that for $n$ large enough:
    \begin{enumerate}
        \item $z_{n,i} \neq z_{n,j}$ for $i \neq j$
        \item $\det \mT_n(z_{n,i}) = 0$ for $i = 1, \ldots, l$.
        \item $z_{n,i} \to 
                z_0 
                = 
                m^{-1} \left( \frac{1}{\mu + \gamma^{-1}} \right)$.
    \end{enumerate}
\end{lemma}
\noindent
The proof involves a technical lemma, which we state and prove now.

\begin{lemma}\label{L:uniform-converge-same-roots}
    Let $g_n(z)$ be a sequence of continuous real-valued functions that 
    converge uniformly on $(u,v)$ to $g_0(z)$.  If $g_0(z)$ is analytic
    on $(u,v)$, then for $n$ large enough, $g_n(z)$ and $g_0(z)$ have the
    same number of zeros in $(u,v)$ (counting multiplicity).
\end{lemma}
\begin{proof}
    Since $|g_0(z)|$ is bounded away from zero outside the neighborhoods
    of its zeros, we can assume without loss of generality that $g_0(z)$ has
    a single zero $z_0 \in (u,v)$ of multiplicity $l$.  Define
    \(
        \tg_n(z) = \frac{g_n(z)}{|z - z_0|^{l-1}}.
    \)
    The function $\tg_0(z)$ is bounded and continuous, and has a 
    simple zero at $z_0$.  For $r$ small enough, $\tg_0(z_0 + r)$ and
    $\tg_0(z_0 - r)$ have differing signs.  Without loss of generality,
    say that the first is positive.  
    
    The sequence $\tg_n(z)$ converges uniformly to $\tg_0(z)$ outside of
    a neighborhood of $z_0$.  Thus, for $n$ large enough, $\tg_n(z-r) < 0$
    and $\tg_n(z+r) > 0$.  Also, either $g_n(z)$ has a zero at $z_0$, or else 
    for a small enough neighborhood around $z_0$, $\sgn \tg_n(z)$ is constant.
    Since $\tg_n(z)$ is continuous outside of a neighborhood of $z_0$, 
    $\tg_n(z)$ and $g_n(z)$ must have a zero
    in $(z_0-r,z_0+r) \subset (u,v)$.  Call this zero $z_{n,1}$.  Since
    $r$ is arbitrary, $z_{n,1} \to z_0$, and
    $\frac{g_n(z)}{z - z_{n,1}} \to \frac{g_0(z)}{z - z_0}$ uniformly on
    $(u,v)$.  We can now proceed inductively, since $\frac{g_0(z)}{z - z_0}$
    has a zero of multiplicity $l-1$ at $z_0$.
\end{proof}

\begin{proof}[Proof of Lemma~\ref{L:zn-sequence-exists}]
    Let $z_0$ be the unique solution of $m(z_0) = (\mu + \gamma^{-1})^{-1}$.
    Define $g_n(z) = \det \mT_n(z)$.  Since $\det$ is a continuous function, 
    $g_n(z)$ converges uniformly to $g_0(z)$ in 
    any neighborhood of $z_0$.  Noting that $g_0(z)$ has a zero of 
    multiplicity 
    $l$ at $z_0$, by Lemma~\ref{L:uniform-converge-same-roots} we get that for large enough $n$, $g_n(z)$ has $l$ zeros
    in a neighborhood of $z_0$.  By a lemma of Okamoto~\cite{okamoto1973deq}, 
    the zeros of $g_n(z)$ are almost surely simple.
\end{proof}

The last thing we need to do is show that for each sequence
$z_{n,i}$ solving the equation $\det \mT_n(z_{n,i}) = 0$, there is a corresponding sequence
of vectors $\vx_{n,i}$ with $f_n(z_{n,i}, \vx_{n,i}) = 0$.  Since $\det \mT_n (z_{n,i}) = 0$, there exists an $\vx_{n,i}$ with 
$\mT_{n} (z_{n,i}) \, \vx_{n,i} = 0$.  We need to show that the sequence of vectors has a limit.

Every solution pair $(z_{n,i}, \vx_{n,i})$ determines a unique
eigenvalue-eigenvector pair through equation~\eqref{E:v-from-v1}.  Since the eigenvalues of $\mS_n$ are almost surely unique, with
the identifiability restriction of \eqref{E:x-identifiability} we must have that $z_{n,i}$ uniquely determines $\vx_{n,i}$.  Suppose that $z_{n,i}$ is the $i$th largest solution of $\det \mT_n(z) = 0$, and that $f_n( z_{n,i}, \vx_{n,i}) = 0$.  We will now show that $\vx_{n,i} \toas \ve_i$.

\begin{lemma}\label{L:xn-converges}
    Suppose that $f_n(z_{n,i}, \vx_{n,i}) = 0$, that $\vx_{n,i}$ satisfies the 
    identifiability restriction \eqref{E:x-identifiability}, and that
    $z_{n,i} \toas \bmu_i$.
    If $\mu_i \neq \mu_j$ for all $j \neq i$, then $\vx_{n,i} \toas \ve_i$.
\end{lemma}

We will use a perturbation lemma, which follows from the $\sin \Theta$ theorem (see Stewart and Sun~\cite{stewart1990mpt}[p. 258]).

\begin{lemma}\label{L:approx-eigenpair}
    Let $(z,\vx)$ be an approximate eigenpair of the $k\times k$
    matrix $\mA$ (in the sense that $\mA \vx \approx z \vx$), with 
    $\| \vx \|_2 = 1$.  Let $\vr = \mA \vx - z \vx$.
    Suppose that there is a set $\mathcal{L}$ of $k-1$ eigenvalues of $\mA$ 
    such that
    \[
        \delta = \min_{l \in \mathcal{L}} | l - z | > 0.
    \]
    Then there is an eigenpair $(z_0, \vx_0)$ of $\mA$ with 
    $\| \vx_0 \|_2 = 1$ satisfying
    \[
        \vx^\trans \vx_0
            \geq
                \sqrt{ 1 - \frac{\|\vr\|_2^2}{ \delta^2 } }
    \]
    and
    \[
        | z - z_0 |
            \leq \| \vr \|_2.
    \]
\end{lemma}

\begin{proof}[Proof of Lemma~\ref{L:xn-converges}]
    We have that $z_{n,i} \toas \bmu_i$ and that
    $\mT_n(z_{n,i}) \vx_{n,i} = 0$.  Since
    $\mT_n(z_{n,i}) \toas \mT_0(\bmu_i)$, we get
    \begin{align*}
        \| \mT_0(\bmu_i) \vx_{n,i} \|_2
            &= \| \big(\mT_n(z_{n,i} - \mT_0(\bmu_i) \big) \vx_{n,i} \|_2 \\
            &\leq \| \mT_n(z_{n,i} - \mT_0(\bmu_i) \|_F \\
            &\toas 0.
    \end{align*}
    Since $\mu_i$ is distinct, $0$ is a simple eigenvalue of $\mT_0(\bmu_i)$.
    Thus, all other eigenvalues have magnitude at least $\delta > 0$ for some 
    $\delta$.
    Define $\vr_n = \mT_0(\bmu_i) \vx_{n,i}$.  By 
    Lemma~\ref{L:approx-eigenpair}, there exists an eigenpair $(z_0, \vx_0)$
    of $\mT_0(\bmu_i)$ with $| z_0 | \leq \| \vr_n \|_2$ and
    $ \vx_{n,i}^\trans \, \vx_0 \geq
        \sqrt{ 1 - \frac{\|\vr\|_2^2}{ \delta^2 } }$.  Since 
    $\|\vr_n\| \toas 0$, for $n$ large enough we must have $z_0 = 0$ and
    $\vx_0 = \ve_i$ or $-\ve_i$.  Lastly, noting that $\vx_0$ and $\vx_{n,i}$
    are both unit vectors, we get
    \begin{align*}
        \| \vx_{n,i} - \vx_0 \|_2^2
            &= 2 - 2 \vx_{n,i}^\trans \, \vx_0 \\
            &\toas 0.
    \end{align*}
    With the identifiability restriction, this forces $\vx_{n,i} \toas \ve_i$.
\end{proof}

Finally, we show that eventually the points described in Lemma~\ref{L:zn-sequence-exists} are the only zeros of $f_n(z, \vx)$ having
$z > b_\gamma$.  Since $f_n(z, \vx) = 0$ implies 
$\det \mT_n(z) = 0$, it suffices to show that $\mT_n(z)$ has no other
zeros.

\begin{lemma}\label{L:detT-solution-upper-bound}
    For $n$ large enough, almost surely
    the equation $\det \mT_n(z) = 0$ has exactly 
    $\bar k$ solutions in  $\big( b_\gamma, \infty \big)$
    (namely, the $\bar k$  points described in 
    Lemma~\ref{L:zn-sequence-exists}).
\end{lemma}
\begin{proof}
    By Lemma~\ref{L:uniform-converge-same-roots}, for $n$ large enough
    $\det \mT_n(z)$ and $\det \mT_0(z)$ have the same number of solutions in 
    $(u,v) \subset \big( b_\gamma, \infty \big)$.  Thus, we only need to
    show that the solutions of $\det \mT_n(z)$ are bounded.  Since every
    solution is an eigenvalue of $\mS_n$, this amounts to showing that the
    eigenvalues of $\mS_n$ are bounded.  Using the Courant-Fischer 
    min-max characterization of eigenvalues \cite{golub1996mc}, we have
    \begin{align*}
        \| \mS_n \|_2
            &= \frac{1}{\sqrt{n}} \| \mX_n \|_2^2 \\
            &\leq 
                \left(
                    \| \mU_n \mD_n \mV_n^\trans \|_2
                    +
                    \frac{1}{\sqrt{n}}
                    \| \mE_n \|_2
                \right)^2 \\
            &\toas
                \left( \sqrt{\mu_1} + \sqrt{b_\gamma} \right)^2.
    \end{align*}
    Thus, the solutions of $\det \mT_n(z) = 0$ are almost surely bounded.
\end{proof}


\subsection{Second-order behavior}

To find the second-order behavior of the solutions to the secular equation, we use a Taylor-series expansion of $f_n(z,\vx)$ around the limit points.  That is, if $(z_n, \vx_n) \to (z_0, \vx_0)$, we let $D f_n$ be the derivative of $f_n$ and write
\[
    0
    =
    f_n (z_n, \vx_n)
        \approx
        f_n( z_0, \vx_0 )
        + 
        D f_n(z_0, \vx_0)
        \left(
        \begin{matrix}
            z_n   - z_0 \\
            \vx_n - \vx_0
        \end{matrix}
        \right).
\]
We now want to solve for $z_n - z_0$ and $\vx_n - \vx_0$.  Without the identifiability constraint, there are $k$ equations and $k+1$ unknowns, but as soon as we impose the condition $\| \vx_n \|_2 = \| \vx_0 \|_2 = 1$, the
system becomes well-determined.

To make this precise, we first compute
\begin{equation}
    D f_n(z, \vx)
    =
    \left(
    \begin{matrix}
        \mT_0'(z) \vx & \mT_0(z)
    \end{matrix}
    \right)
    +
    \OhP\left( \frac{1}{\sqrt{n}} \right),
\end{equation}
with pointwise convergence in $z$.  Then, we write
\[
    f_n( z_n, \vx_n)
        = 
        f_n (z_0, \vx_0)
        +
        D f_n( z_0, \vx_0 )
        \left(
        \begin{matrix}
            z_n - z_0 \\
            \vx_n - \vx_0
        \end{matrix}
        \right)
        +
        \OhP \left(
            (z_n - z_0)^2
            +
            \| \vx_n - \vx_0 \|_2^2
        \right).
\]
If $f_n(z_n, \vx_n) = 0$ and $f_0(z_0, \vx_0) = 0$, then we get
\[
    0
    =
    \frac{1}{\sqrt{n}} f_{n,1} (z_0, \vx_0)
        +
        D f_n( z_0, \vx_0 )
        \left(
        \begin{matrix}
            z_n - z_0 \\
            \vx_n - \vx_0
        \end{matrix}
        \right)
        +
        \OhP \left(
            (z_n - z_0)^2
            +
            \| \vx_n - \vx_0 \|_2^2
        \right).
\]
If $(z_n, \vx_n) \toas (z_0, \vx_0)$, then 
the differences $z_n - z_0$ and $\vx_n - \vx_0$ must be of size
$\OhP\left( \frac{1}{\sqrt{n}} \right)$ and the error term in the Taylor
expanson is size $\OhP\left( \frac{1}{n} \right)$.  The final simplification
we can make is from the length constraint on $\vx_n$ and $\vx_0$.  We have
\begin{align*}
    1 &= \vx_n^\trans \vx_n \\
      &= \big( \vx_0 + (\vx_n - \vx_0) \big)^\trans
         \big( \vx_0 + (\vx_n - \vx_0) \big) \\
      &= 1 + 2 \vx_0^\trans ( \vx_n - \vx_0 ) + \| \vx_n - \vx_0 \|_2^2,
\end{align*}
so that $\vx_0^\trans ( \vx_n - \vx_0 ) = \OhP\left( \frac{1}{n} \right)$.  With a little more effort, we can solve for $z_n - z_0$ and $\vx_n - \vx_0$.

\begin{lemma}
    If $(z_n, \vx_n)$ is a sequence converging almost surely to 
    $(\bmu_i, \ve_i)$, such that $f_n(z_n, \vx_n) = 0$ and $\mu_i \neq \mu_j$
    for $i \neq j$, then:
    \begin{enumerate}[(i)]
        \item \begin{equation}
                  \sqrt{n} ( z_n - \bmu_i) 
                  = 
                  -
                  \frac{\big(\mT_{n,1}(\bmu_i) \big)_{ii}}
                       {\big(\mT_0'(\bmu_i) \big)_{ii}}
                  +
                  \ohP(1),
              \end{equation}
        \item \begin{equation}
                   \sqrt{n} \left( x_{n,i} - 1 \right)
                   =
                   \ohP\left( 1 \right), \quad \text{and}
              \end{equation}
        \item \begin{equation}
                  \sqrt{n} \, x_{n,j}
                  = 
                  -
                  \frac{\big(\mT_{n,1}(\bmu_i) \big)_{ji}}
                       {\big(\mT_0(\bmu_i) \big)_{jj}}
                  +
                  \ohP(1)
                  \quad
                  \text{for $i \neq j$.}
              \end{equation}
    \end{enumerate}
    \begin{proof}
        We have done most of the work in the exposition above.  In particular,
        we already know that ($ii$) holds.  Using the Taylor expansion, we 
        have
        \[
            0
            =
            \frac{1}{\sqrt{n}}
            \mT_{n,1}(\bmu_i) \, \ve_i
            +
            \left(
            \begin{matrix}
                \mT_0'(\bmu_i) \, \ve_i & \mT_0(\bmu_i)
            \end{matrix}
            \right)
            \left(
            \begin{matrix}
                z_n - \bmu_i \\
                \vx_n - \ve_i
            \end{matrix}
            \right)
            +
            \ohP\left( \frac{1}{\sqrt{n}} \right).
        \]
        Since $\mT_0$ and $\mT_0'$ are diagonal, using ($ii$) we get
        \begin{align*}
            0
                &=
                \frac{1}{\sqrt{n}}
                \big(\mT_{n,1}(\bmu_i) \big)_{ii}
                +
                \big(\mT_0'(\bmu_i) \big)_{ii} ( z_n - \bmu_i )
                +
                \ohP\left( \frac{1}{\sqrt{n}} \right), \\
            0
                &=
                \frac{1}{\sqrt{n}}            
                \big(\mT_{n,1}(\bmu_i) \big)_{ji}
                +
                \big(\mT_0(\bmu_i) \big)_{\!jj} \,
                x_{n,j} 
                +
                \ohP\left( \frac{1}{\sqrt{n}} \right)
                \qquad \text{for $i \neq j$.}
        \end{align*}
        Therefore, ($i$) and ($iii$) follow.
    \end{proof}
\end{lemma}

At this point, it behooves us to do some simplification.  For $\mu_i, \mu_j > \gamma^{-1/2}$, we have
\[
    m(\bmu_i) = - \frac{1}{\mu_i + \gamma^{-1}}.
\]
Using \eqref{E:stieltjes-inverse}, this implies
\begin{align*}
    \bmu_i
        &= \mu_i + 1 + \gamma^{-1} + \frac{ \gamma^{-1} }{ \mu_i } \\
        &= \Big( \mu_i + 1 \Big)
           \left( \frac{ \mu_i + \gamma^{-1} }{ \mu_i } \right).
\end{align*}
Note that this agrees with the definition of $\bmu_i$ in Theorem~\ref{T:spiked-eigenvalue-limits}.  Now,
\[
    \frac{ m(\bmu_i) - m(\bmu_j) }
         { \bmu_i - \bmu_j }
    =
    \frac{1}{ ( \mu_i + \gamma^{-1} ) ( \mu_j + \gamma^{-1} ) }
    \cdot
    \frac{ \mu_i \mu_j }{ \mu_i \mu_j - \gamma^{-1} },
\]
so that
\[
    m'(\bmu_i)
    =
    \frac{ 1 }{ ( \mu_i + \gamma^{-1} )^2 }
    \cdot
    \frac{ \mu_i^2 }{ \mu_i^2 - \gamma^{-1} }.
\]
Also,
\[
    \frac{ \bmu_i \, m( \bmu_i ) - \bmu_j \, m( \bmu_j ) }
         { \bmu_i - \bmu_j }
        =
            \frac{1}{\mu_i \, \mu_j - \gamma^{-1}}.
\]
We can compute
\begin{align*}
    \big( \mT_0(\bmu_i) \big)_{jj}
        &=
            ( \mu_i - \mu_j )
            \left( \frac{ \mu_i + \gamma^{-1} }{ \mu_i } \right) 
                \qquad\text{ for $j \neq i$,} \\
    \big( \mT_0'(\bmu_i) \big)_{ii}
        &=
            -
            \left(
                \frac{ \mu_i^2 }{ \mu_i^2 - \gamma^{-1} }
            \right)
            \left(
                \frac{\mu_i + \gamma^{-1}}{ \mu_i }
            \right).
\end{align*}
Since
\(
    \big( 1 + \gamma^{-1} m( \bmu_i ) \big)^{-1}
        = \frac{ \mu_i + \gamma^{-1} }{ \mu_i },
\)
we get
\[
    \begin{split}
        \mT_{n,1}(\bmu_i)
           &=
                \left(
                    \frac{1}{\sqrt{n}}
                    \mE_{n,31}^\trans \mE_{n,31}
                    -
                    \sqrt{n}
                    (1 - \gamma^{-1})
                    \mI_k
                \right) \\
            &\quad-
                \left(
                    \frac{\mu_i + \gamma^{-1}}{\mu_i}
                \right)^2
                \mD \mF_n (\bmu_i ) \mD \\
            &
                \begin{split}
                \quad+
                \left(
                    \frac{\mu_i + \gamma^{-1}}{\mu_i}
                \right)
                \Big(
                    \sqrt{n} ( \mD_n^2 - \mD )
                    &+
                    \mD
                    \big( \mE_{n,11} - \mG_{n} (\bmu_i) \big) \\
                    &+
                    \big( \mE_{n,11} - \mG_{n} (\bmu_i) \big)^\trans                    
                    \mD
                \Big) 
                \end{split}\\
            &\quad-
                (\mu_i + 1)
                \left(
                    \frac{\mu_i + \gamma^{-1}}{\mu_i}
                \right)
                \mH_n(\bmu_i).
    \end{split}
\]
The first term converges in distribution to a mean-zero multivariate normal
with variance $2 \, (1 - \gamma^{-1})$ along the diagonal and variance $1 - \gamma^{-1}$ otherwise; the elements
are all uncorrelated except for the obvious symmetry.  Also, we have
\begin{align*}
    \cov\big( F_{ij} ( \bmu_1 ), F_{ij} ( \bmu_2 ) \big)
        &= 
        \gamma^{-1} ( 1 + \delta_{ij} )
        \cdot
        \frac{1}{ ( \mu_1 + \gamma^{-1} ) ( \mu_2 + \gamma^{-1} ) }
        \cdot
        \frac{ \mu_1 \mu_2 }{ \mu_1 \mu_2 - \gamma^{-1} }, \\
    \cov\big( G_{ij} ( \bmu_1 ), G_{ij} ( \bmu_2 ) \big)
        &= 
        \gamma^{-1} 
        \cdot
        \frac{1}{ \mu_1 \mu_2 - \gamma^{-1} }, \\
    \cov\big( H_{ij} ( \bmu_1 ), H_{ij} ( \bmu_2 ) \big)
        &= 
        \gamma^{-1} ( 1 + \delta_{ij} )
        \cdot
        \frac{1}{ ( \mu_1 + \gamma^{-1} ) ( \mu_2 + \gamma^{-1} ) }
        \cdot
        \frac{ \mu_1 \mu_2 }{ \mu_1 \mu_2 - \gamma^{-1} }.
\end{align*}
Therefore, for $j \neq i$ we have variances
\begin{subequations}
\begin{align}
    \begin{split}
    \var \left(
        \big(
            \mT_{n,1}( \bmu_i ) \,
            \ve_i
        \big)_i
    \right)
        &= 
        \sigma_{ii}
        \cdot
        \left(
            \frac{\mu_i + \gamma^{-1}}{\mu_i}
        \right)^2 \\
        &\qquad+
        2
        \left(
            2 \mu_i + 1 + \gamma^{-1}
        \right)
        \frac{ (\mu_i + \gamma^{-1})^2 }
             { \mu_i^2 - \gamma^{-1} } + \oh(1), 
    \end{split} \\
    \var \left(
        \big(
            \mT_{n,1}( \bmu_i ) \,
            \ve_i
        \big)_j
    \right)
        &=
        \left(
            2 \mu_i + 1 + \gamma^{-1}
        \right)
        \frac{ (\mu_i + \gamma^{-1})^2 }
             { \mu_i^2 - \gamma^{-1} } + \oh(1),
\end{align}
\end{subequations}
and nontrivial covariances
\begin{subequations}
\begin{align}
    \cov \left( 
        \big(
            \mT_{n,1}( \bmu_i ) \,
            \ve_i
        \big)_i, \,
        \big(
            \mT_{n,1}( \bmu_j ) \,
            \ve_j
        \big)_j
    \right)
        &=
        \sigma_{ij}
        \cdot
        \frac{\mu_i + \gamma^{-1}}{ \mu_i }
        \cdot
        \frac{\mu_j + \gamma^{-1}}{ \mu_j } + \oh(1) \\
\intertext{and}
    \begin{split}
    \cov \left(
        \big(
            \mT_{n,1}( \bmu_i ) \,
            \ve_i
        \big)_j, \,
        \big(
            \mT_{n,1}( \bmu_j ) \,
            \ve_j
        \big)_i    
    \right)
        &=
        \left(
            \mu_i + \mu_j + 1 + \gamma^{-1}
        \right) \\
        &\qquad \cdot
            \frac{ ( \mu_i + \gamma^{-1} )( \mu_j + \gamma^{-1} ) }
                 { \mu_i \mu_j - \gamma^{-1} }
        + \oh(1).
    \end{split}
\end{align}
\end{subequations}
All other covariances between the elements of
$\mT_{n,1}(\bmu_i) \, \ve_i$ and $\mT_{n,1}(\bmu_j) \, \ve_j$ are zero.

\section{Singular values and singular vectors}\label{S:singular-values-and-vectors}

The results about solutions to the secular equation translate directly
to results about the singular values and right singular vectors of
$\frac{1}{\sqrt{n}} \mtX_n$.

\subsection{Singular values}

Every value $z$ with $f_n(z, \vx) = 0$ for some $\vx$ is the square of a
singular value of $\frac{1}{\sqrt{n}} \mtX_n$.  Therefore,  Section~\ref{S:solutions-to-secular-eq} describes the behavior of the top $\bar k$ singular values.  To complete the proof of Theorem~\ref{T:spiked-eigenvalue-limits} for $\gamma \geq 1$, we only need to describe what happens to the singular values corresponding to the indices $i$ with $\mu_i \leq \gamma^{-1/2}$.

\begin{lemma}\label{L:eigenvalue-below-threshold-limit}
    If $\mu_i \leq \gamma^{-1/2}$ then the $i$th eigenvalue of
    $\mS_n$, converges almost surely to $b_\gamma$.
\end{lemma}
\begin{proof}
From Lemma~\ref{L:detT-solution-upper-bound}, we know that for $n$ large
enough and $\epsilon$ small, there are exactly
$\bar k = \max \{ i : \mu_i > \gamma^{-1/2} \}$ eigenvalues of $\mS_n$
in $\big( b_\gamma + \epsilon, \infty)$.  From the eigenvalue interleaving
inequalities, we know that the $i$th eigenvalue of $\mS_n$ is at least as
big as the $i$th eigenvalue of $\mS_{n,22}$.  

Denote by $\hmu_{n,i}$ the $i$th eigenvalue of $\mS_n$, with
$\bar k < i \leq k$.  Then almost surely,
\[
    \lim_{n\to\infty} \lambda_{n,i} 
        \leq
        \liminf_{n\to\infty} \hmu_{n,i}
        \leq
        \limsup_{n\to\infty} \hmu_{n,i}        
        \leq
        b_\gamma + \epsilon.
\]
Since $\lambda_{n,i} \toas b_\gamma$ and $\epsilon$ is arbitrary, this
forces $\hmu_{n,i} \toas b_\gamma$.
\end{proof}

\begin{lemma}
    If $\mu_i \leq \gamma^{-1/2}$, then
    \(
        \sqrt{n} ( \hmu_i - \bmu_i ) \toP 0,
    \)
    where $\hmu_i$ is the $i$th eigenvalue of $\mS_n$.
\end{lemma}
\begin{proof}
We use the same notation as in Lemma~\ref{L:eigenvalue-below-threshold-limit}.
Recall that in the present situation, $\bmu_i = b_\gamma$.
Since $\lambda_{n,i} = b_\gamma + \OhP(n^{-2/3})$, we have that
\[
    \sqrt{n} \big( \lambda_{n,i} - b_\gamma \big) \toP 0.
\]
This means that
\[
    \liminf_{n\to\infty} \sqrt{n} \big( \hmu_{n,i} - b_\gamma \big)
        \geq \ohP(1).
\]
The upper bound is a little more delicate.  By the Courant-Fischer min-max
characterization, $\hmu_i^{1/2}$ is bounded above by $\tmu_i^{1/2}$, the 
$i$th singular value of
\[
    \frac{1}{\sqrt{n}} \mX_n 
    + 
    \sqrt{n} \big( (\gamma^{-1/2} + \epsilon)^{1/2} - \mu_i^{1/2} \big)
    \vu_{n,i} \vv_{n,i}^\trans
\]
for any $\epsilon > 0$.  From the work in  Section~\ref{S:solutions-to-secular-eq}, we know that
\begin{align*}
    \tmu_i
        &=
            \left(
                1 + \gamma^{-1/2} + \epsilon
            \right)
            \left(
                1
                +
                \frac{ 1 }
                     { \gamma^{1/2} + \gamma \epsilon }
            \right)
            +
            \OhP \left(
                \frac{\tsigma_i}{ \sqrt{n} }
            \right) \\
        &=
            b_\gamma
            +
            \epsilon^2
            \frac{ \gamma^{1/2} - 1 }{ 1 + \gamma^{1/2} \epsilon }
            +
            \OhP \left(
                \frac{\tsigma_i}{ \sqrt{n} }
            \right),
\end{align*}
where
\begin{align*}
    \begin{split}
    \tsigma_i^2
    &=
    \sigma_{ii}
    \left(
        1 - \frac{1}{1 + 2 \gamma^{1/2} \epsilon + \gamma \epsilon^2}
    \right)^2 \\
    &\quad+
    2\big( 2 (\gamma^{-1/2} + \epsilon)  + 1 + \gamma^{-1} \big)
    \left(
        1 - \frac{1}{1 + 2 \gamma^{1/2} \epsilon + \gamma \epsilon^2}    
    \right)
    \end{split} \\
    &= \Oh( \epsilon ).
\end{align*}
Therefore, for all $0 < \epsilon < 1$, we have
\[
    \sqrt{n}
    \left(
        \hmu_{n,i}
        - 
        b_\gamma 
        - \epsilon^2
        \frac{ \gamma^{1/2} - 1 }{ 1 + \gamma^{1/2} \epsilon }
    \right)
    \leq
    \OhP \left( \epsilon^{1/2} \right).
\]
Letting $\epsilon \to 0$, we get
\[
    \limsup_{n\to\infty} 
        \sqrt{n}
        \big(
            \hmu_{n,i}
            -
            b_\gamma
        \big)
        \leq
        \ohP(1).
\]
Together with the lower bound, this implies
\(
    \sqrt{n}
    \big(
        \hmu_{n,i}
        -
        b_\gamma
    \big)
    \toP
    0.
\)
\end{proof}

\subsection{Right singular vectors}

For $\mu_i > \gamma^{-1/2}$, we can get a right singular vector of $\frac{1}{\sqrt{n}} \mtX_n$ from the sequence of solution pairs 
$(z_{n,i}, \vx_{n,i})$ satisfiying 
$f_n(z_{n,i}, \vx_{n,i}) = 0$ and $z_{n,i} \toas \bmu_i$.  The vector
is parallel to
\begin{equation}
    \vtx_{n,i}
    =
    \left(
    \begin{matrix}
        \vx_n \\
        - ( \mS_{n,22} - z_n \mI_{p-k} )^{-1} \mS_{n,21} \, \vx_n
    \end{matrix}
    \right).
\end{equation}
We just need to normalize this vector to have unit length.  The length
of $\vtx_n$ is given by
\begin{equation}
    \| \vtx_n \|_2^2
        =
        \vx_n^\trans
        \left(
            \mI_{k}
            +
            \mS_{n,12}
            ( \mS_{n,22} - z_n \mI_{p-k} )^{-2} 
            \mS_{n,21}
        \right)
        \vx_n.
\end{equation}
It is straightforward to show that for $z > b_\gamma$,
\begin{align*}
    \mI_{k}
    +
    \mS_{n,12}
    ( \mS_{n,22} - z \mI_{p-k} )^{-2} 
    \mS_{n,21}
        &\toas
            -\mT_0'(z) \\
        &= \frac{ \gamma^{-1} m'(z) }
                { \big( 1 + \gamma^{-1} m(z) \big)^2 }
           \mD^2
           +
           \frac{ m'(z) }{ \big( m(z) \big)^2 }
           \mI_k
\end{align*}
uniformly for $z$ in any compact subset of $\big( b_\gamma, \infty \big)$.
It is also not hard to compute for $\mu_i > \gamma^{-1/2}$ that
\[
    -\mT_0'( \bmu_i )
        =
            \frac{ \gamma^{-1} }{ \mu_i^2 - \gamma^{-1} }
            \mD^2
            +
            \frac{ \mu_i^2 }{ \mu_i^2 - \gamma^{-1} }
            \mI_k.
\]
Therefore, if $\mu_i > \gamma^{-1/2}$ and 
$(z_{n,i}, \vx_{n,i}) \toas (\bmu_i, \ve_i)$, then
\begin{align*}
    \| \vtx_{n,i} \|_2^2
        &\toas
            \frac{ \mu_i ( \mu_i + \gamma^{-1} ) }
                 { \mu_i^2 - \gamma^{-1} } \\
        &=
            \frac{ 1 + \tfrac{1}{ \gamma \mu_i} }
                 { 1 - \tfrac{1}{ \gamma \mu^2} }.
\end{align*}

The behavior of the right singular vectors when $\mu_i \leq \gamma^{1/2}$ is
a little more difficult to get at.  We will use a variant of an argument from Paul~\cite{paul2007ase} to show that $\| \vtx \|_2 \toas \infty$, which implies that $\frac{ \vx_{n,i} }{ \| \vtx_{n,i} \|_2 } \toas 0$.  We can
do this by showing the smallest eigenvalue of 
\(
    \mS_{n,12}
    ( \mS_{n,22} - z_{n,i} \mI_{p-k} )^{-2} 
    \mS_{n,21}
\)
goes to $\infty$.

Write
\[
    \mS_{n,12}
    ( \mS_{n,22} - z_{n,i} \mI_{p-k} )^{-2} 
    \mS_{n,21}
        =
        \sum_{\alpha=1}^{p-k}
            \frac{ \vs_{n,\alpha} \, \vs_{n,\alpha}^\trans }
                 { ( \tlambda_{n,\alpha} - z_{n,i} )^2 },
\]
where $\vs_{n,1}, \vs_{n,2}, \ldots, \vs_{n,p-k}$ are the 
eigenvectors of $\mS_{n,22}$ multiplied by $\mS_{n,12} = \mS_{n,21}^\trans$, and
$\tlambda_{n,1}, \tlambda_{n,2}, \ldots, \tlambda_{n,p-k}$ are the eigenvalues
of $\mS_{n,22}$.  For $\epsilon > 0$, define the event 
\(
    J_{n}(\epsilon)
    =
    \left\{
        z_{n,i} < b_\gamma + \epsilon
    \right\}.
\)
From the interleaving inequality, we have $z_{n,i} \geq \tlambda_{n,i}$.
With respect to the ordering on positive-definite matrices, we have
\begin{align*}
    \sum_{\alpha=1}^{p-k}
            \frac{ \vs_{n,\alpha} \, \vs_{n,\alpha}^\trans }
                 { ( \tlambda_{n,\alpha} - z_{n,i} )^2 }
        &\succeq
            \sum_{\alpha=i}^{p-k}
                    \frac{ \vs_{n,\alpha} \, \vs_{n,\alpha}^\trans }
                         { ( \tlambda_{n,\alpha} - z_{n,i} )^2 } \\
        &\succeq
            \sum_{\alpha=i}^{p-k}
                    \frac{ \vs_{n,\alpha} \, \vs_{n,\alpha}^\trans }
                         { ( b_\gamma + \epsilon - z_{n,i} )^2 }
            \qquad\text{on $J_n(\epsilon)$.}
\end{align*}
It is not hard to show that
\(
    \left\|
        \sum_{\alpha=1}^{i-1}
                \frac{ \vs_{n,\alpha} \, \vs_{n,\alpha}^\trans }
                     { ( b_\gamma + \epsilon - z_{n,i} )^2 }
    \right\|
        \toas 0.
\)
Therefore,
\[
    \sum_{\alpha=i}^{p-k}
            \frac{ \vs_{n,\alpha} \, \vs_{n,\alpha}^\trans }
                 { ( b_\gamma + \epsilon - z_{n,i} )^2 }
        \toas
        \frac{ \gamma^{-1} m'\big( b_\gamma + \epsilon \big) }
             { 1 + \gamma^{-1} m\big( b_\gamma + \epsilon \big)}
        \mD^2
        +
        \frac{ m'\big( b_\gamma + \epsilon \big) }
             { \left[ m\big( b_\gamma + \epsilon \big) \right]^2}
        \mI_k.
\]
As $n\to\infty$, we have $\Prob\big( J_n(\epsilon) \big) \to 1$.  So,
since $m'\big( b_\gamma + \epsilon \big) \geq \frac{C}{\sqrt{\epsilon}}$
for some constant $C$, 
letting $\epsilon \to 0$ we must have that the smallest eigenvalue of
\(
    \mS_{n,12}
    ( \mS_{n,22} - z_{n,i} \mI_{p-k} )^{-2} 
    \mS_{n,21}
\)
goes to $\infty$.

\subsection{Left singular vectors}

We can get the left singular vectors from the right from multiplication
by $\frac{1}{\sqrt{n}} \mtX_n$.  Specifically, if $\vtv_{n,i}$ is a right
singular vector of $\frac{1}{\sqrt{n}} \mtX_n$ with singular value
$z_{n,i}^{1/2}$, then $\vtu_{n,i}$, the corresponding left singular vector,
is defined by
\[
    z_{n,i}^{1/2} \,
    \vtu_{n,i}
        = 
        \frac{1}{\sqrt{n}}
        \mtX_n \,
        \vtv_{n,i}.
\]
We are only interested in the first $k$ components of $\vtu_{n,i}$.  If
\[
    \vtv_{n,i}
    =
    \frac{1}{\| \vtx_{n,i} \|_2 }
    \left(
    \begin{matrix}
        \vx_{n,i} \\
        -( \mS_{n,22} - z_{n,i} \mI_{p-k} )^{-1} \mS_{n,21} \, \vx_{n,i}
    \end{matrix}
    \right),
\]
then these are given by
\(
    \frac{1}{\| \vtx_{n,i} \|_2 } \mR_n(z_{n,i}) \, \vx_{n,i},
\)
where
\[
    \mR_n(z)
    =
    \mD_n 
    + 
    \frac{1}{\sqrt{n}}
    \mE_{n,11}
    - 
    \frac{1}{\sqrt{n}}
    \mE_{n,12}
    \left( \mS_{n,22} - z \mI_{p-k} \right)^{-1}
    \mS_{n,21}.
\]
It is not hard to show that
\(
    \mR_n(z) \toas \mR_0(z),
\)
uniformly for $z > b_\gamma$, and
\(
    \mR_n(z) 
        = 
            \mR_0(z) 
            + 
            \frac{1}{\sqrt{n}} \mR_{n,1}(z) 
            + 
            \ohP\left( \frac{1}{\sqrt{n}} \right)
\)
pointwise for $z > b_\gamma$.  Here,
\[
    \mR_0(z) = \frac{1}{ 1 + \gamma^{-1} m(z) } \mD,
\]
and
\[
    \begin{split}
    \mR_{n,1}(z)
    &=
    \big( 1 + \gamma^{-1} m(z) \big)^{-1}
    \Big(
        \sqrt{n} \left( \mD_n  - \mD \right)
        +
        \mE_{n,11}
    \Big) \\
    &\quad- 
    \big( 1 + \gamma^{-1} m(z) \big)^{-2}
    \mF_{n}(z) \mD
    - 
    \mG_n(z).
    \end{split}
\]
Let $y_{n,i} = \frac{1}{\| \vtx_{n,i} \|_2 } \mR_n(z_{n,i}) \, \vx_{n,i}$.
A straightforward calculation shows the following:

\begin{lemma}
    If $\mu_i > \gamma^{-1/2}$ and 
    $(z_{n,i}, \vx_{n,i}) \toas (\bmu_i, \ve_i)$, then 
    \[
        \vy_{n,i}
            \toas
                \left(
                \frac{1 - \tfrac{ 1 }{ \gamma \mu_i^2 } }
                     { 1 + \tfrac{1}{\mu_i} } \right)^{1/2} \,
                \ve_i .
    \]
    If $\mu_i \leq \gamma^{-1/2}$ and $z_{n,i} \toas b_\gamma$, then
    \[
        \vy_{n,i} \toas 0.
    \]
\end{lemma}

\section{Results for $\gamma \in (0,1)$}\label{S:lowrank-gamma-lt-1}

Remarkably the formulas for the limiting quantities still hold when $\gamma < 1$.  To see this, we can get the behavior for $\gamma < 1$ by taking the transpose of $\mX_n$ and applying the theorem for $\gamma \geq 1$.  We switch the roles of $n$ and $p$, replace $\mu_i$ by $\mu_i' = \gamma \mu_i$, and replace $\gamma$ by $\gamma' = \gamma^{-1}$.  Then, for instance, the critical cutoff becomes
\begin{align*}
    \mu_i' &> \frac{1}{\sqrt{\gamma'}}, \quad\text{i.e.} \\
    \gamma \mu_i &> \gamma^{1/2},
\end{align*}
which is the same formula for $\gamma \geq 1$.  The almost-sure limit of
the square of the $i$th eigenvalue of $\frac{1}{p} \mX_n \mX_n^T$ becomes
\begin{align*}
    \bmu_i'
        &= ( \gamma \mu_i + 1 )
           \left( 1 + \frac{\gamma}{ \gamma \mu_i } \right) \\
        &= \gamma ( \mu_i + 1 ) \left( 1 + \frac{1}{\gamma \mu} \right) \\
        &= \gamma \bmu_i.
\end{align*}
The formulas for the other quantities also still remain true.

\section{Related work, extensions, and future work}\label{S:lowrank-related-extensions}

With the proofs completed, we now discuss some extensions and related work.

\subsection{Related work}

When Johnstone~\cite{johnstone2001dle} worked out the distribution of the largest eigenvalue in the null (no signal) case, he proposed studying ``spiked'' alternative models.  Spiked data consists of vector-valued observations with population covariance of the form
\[
    \Sigma = \diag( \mu_1, \mu_2, \ldots, \mu_k, 1, \ldots, 1 ).
\]
Work on the spiked model started with Baik et al.~\cite{baik2005ptl}, 
Baik \& Silverstein~\cite{baik2006els}, and Paul~\cite{paul2007ase}.  Baik et al. showed that for complex Gaussian data, a phase transition phenomenon exists, depending on the relative magnitude of the spike.  Baik \& Silverstein gave the almost-sure limits of the eigenvalues from the sample covariance matrix without assuming Gaussianity.  Paul worked with real Gaussian data and gave the limiting distributions when $\gamma > 1$.  Paul also gives some results about the eigenvectors.

After this initial work, Bai \& Yao~\cite{bai2008clt}~\cite{bai2008lts} derived the almost-sure limits and prove a central limit theorem for eigenvalues for a general class of data that includes colored noise and non-Gaussianity.  Chen et al~\cite{chen2009ppc} consider another type of spiked model with correlation.  Nadler~\cite{nadler2008fsa} derived the behavior of the first eigenvector in a spiked model with one spike.

The above authors all consider data of the form $\mX = \mZ \mSigma^{1/2}$.
Onatski~\cite{onatski2009}, like us, examines data of the form 
$\mX = \mU \mD \mV^T + \mE$.  With slightly different assumptions than ours, he is able to give the probability limits of the top eigenvalues, along with the marginal distributions of the scaled eigenvalues and singular vectors.  However, Onatski does not work in a ``transpose-agnostic'' framework like we do, so his methods do not allow getting at the joint distribution of the left and right singular vectors.

\subsection{Extensions and future work}

We have stopped short of computing the second-order behavior of the singular vectors, but no additional theory is required to get at these quantities.  Anyone patient enough can use our results to compute the joint distribution of 
$\| \vtx_{n,i} \|_2$, $\vx_{n,i}$, and $\vy_{n,i}$.  This in turn will give the joint behavior of the singular vectors.  

Most of the proof remains unchanged for complex Gaussian noise, provided
transpose ($^\trans$) is replaced by conjugate-transpose ($^\herm$).  The
variance formulas need a small modification, since the fourth-moment of
a real Gaussian is $3$ and that of a complex Gaussian is $2$.

For colored or non-Gaussian noise, we no longer have orthogonal invariance, so the change of basis in Section~\ref{S:lowrank-preliminaries} is a little trickier.  It is likely that comparable results can still be found, perhaps using results on the eigenvectors of general sample covariance matrices from Bai et al. \cite{bai2007ael}.


multivariate.tex

\chapter{Multivariate statistics background}\label{C:multivariate-background}

Multivariate statistics will prove to be a central tool for this thesis.  We use this chapter to gather the relevant definitions and results, concentrating mainly on the eigenvalues and eigenvectors from sample covariance matrices.  The literature in this area spans over fifty years.  We give the basic definitions and properties of the multivariate normal and Wishart distributions in Section~\ref{S:mulitivariate-definitions}.  Then, in Section~\ref{S:multivariate-classical}, we survey classical results about sample covariance matrices when the number of dimensions, $p$, is fixed and the sample size, $n$, grows to infinity.  This material is by now standard and can be found in any good multivariate statistics book (e.g. Muirhead~\cite{muirhead1982ams}).  Lastly,  in Section~\ref{S:multivariate-modern}, we survey modern asymptotics, where $n \to \infty$ and $p$ grows with $n$.  Modern multivariate asymptotics is still an active research topic, but today it is possible to give a reasonably-complete description of the objects of interest.

\section{The multivariate normal and Wishart distributions}\label{S:mulitivariate-definitions}

We start with the definition of the multivariate normal distribution and some basic properties, which can be found, for example, in Muirhead~\cite{muirhead1982ams}[Chapters 1--3].

\begin{definition}[Multivariate Normal Distribution]
    \label{D:multivariate-normal}
For mean vector 
\(
    \vmu \in \reals^p
\)
and positive-semidefinite covariance matrix
\(
    \mSigma \in \reals^{p\times p},
\)
a random vector 
\(
    \vX \in \reals^p
\)
is said to be distributed from the multivariate normal distribution, denoted 
\(
    \vX 
    \sim 
    \Normal \left(
        \vmu, \,
        \mSigma
    \right)
\)
if for every fixed vector $\va \in \reals^p$, the vector
$\va^\trans \vX$ has a univariate normal distribution with mean
\(
    \va^\trans \vmu
\)
and variance
\(
    \va^\trans \mSigma \va
\).
\end{definition}

\noindent
The multivariate normal distribution is defined for any positive-semidefinite covariance matrix $\mSigma$, but it only has a density when $\mSigma$ is strictly positive-definite.

\begin{proposition}
If $\vX \in \reals^p$ follows a multivariate normal distribution with mean $\vmu$ and positive-definite covariance matrix $\mSigma$, then its components have density
\begin{equation}\label{E:normal-density}
    f( \vx )
    =
    (2 \pi )^{-p/2}
    |\mSigma|^{-1/2}
    \exp\left(
        -
        \frac{1}{2}
        (\vx - \vmu)^\trans
        \mSigma^{-1}
        (\vx - \mu)
    \right).
\end{equation}
\end{proposition}

A basic fact about the multivariate normal is the following:

\begin{proposition}\label{P:scale-shift-normal}
Let 
\(
    \va \in \reals^p,
\)
\(
    \mC \in \reals^{q \times p},
\)
and 
\(
    \vX 
    \sim
    \Normal \left( 
        \vmu, \,
        \mSigma
    \right).
\)
Define $\vY = \mC \vX + \va$.  Then 
\(
    \vY
    \sim
    \Normal \left( 
        \mC \vmu + \va, \,
        \mC \mSigma \mC^\trans
    \right).
\)
\end{proposition}

\noindent
Two immediate corollaries are:

\begin{corollary}\label{C:normal-orthog-invariant}
Suppose that
\(
    \vX 
    \sim 
    \Normal \left( 
        \vzero, \,
        \sigma^2 \mI_p
    \right)
\)
and that
\(
    \mO \in \reals^{p \times p}
\)
is an orthogonal matrix.  Then
\(
    \mO \vX \eqd \vX.
\)
\end{corollary}

\begin{corollary}
If
\(
    \vX 
    \sim 
    \Normal \left( 
        \vzero, \,
        \mSigma
    \right)
\)
and $\mSigma = \mC \mC^\trans$ for a matrix $\mC \in \reals^{p \times p}$,
and if
\(
    \vZ
    \sim
    \Normal \left(
        \vzero, \,
        \mI_p
    \right),
\)
then
\(
    \mC \vZ
    \eqd
    \vX
\).
\end{corollary}    

We are often interested in estimating the underlying parameters from
multivariate normal data.  The sufficient statistics are the standard
estimates.

\begin{proposition}\label{P:normal-sufficient-stats}
Say that
\(
    \vX_1, \vX_2, \ldots, \vX_n
\) 
are independent draws from a
\(
    \Normal \left(
        \vmu, \,
        \mSigma
    \right)
\)
distribution.  Then the sample mean
\begin{equation}\label{E:sample-mean}
    \vbX_n
    \define
    \frac{1}{n}
    \sum_{i=1}^n
        \vX_i
\end{equation}
and the sample covariance
\begin{equation}\label{E:sample-covariance}
    \mS_n
    \define
    \frac{1}{n-1}
    \sum_{i=1}^n
        \left(
            \vX_i - \vbX_n
        \right)
        \left(
            \vX_i - \vbX_n
        \right)^\trans
\end{equation}
are sufficient statistics for $\vmu$ and $\mSigma$.
\end{proposition}

To describe the distribution of $\mS_n$, we need to introduce the 
Wishart distribution.

\begin{definition}[Wishart Distribution]\label{D:wishart}
Let $\vX_1, \vX_2, \ldots, \vX_n \in \reals^p$ be an iid sequence of
random vectors, each distributed as
\(
    \Normal \left(
        \vzero, \,
        \mSigma
    \right).
\)
Then the matrix
\[
    \mA = \sum_{i=1}^n \vX_i \vX_i^\trans
\]
is said to have the Wishart distribution with $n$ degrees of freedom and scale parameter $\mSigma$.  We denote this by
\(
    \mA
    \sim
    \Wishart_p \left(
        n, \,
        \mSigma
    \right).
\)
\end{definition}

\noindent
When $n \geq p$ and $\mSigma$ is positive-definite, the elements of a Wishart matrix have a density.
\begin{proposition}
Suppose that
\(
    \mA
    \sim
    \Wishart_p \left(
        n, \,
        \mSigma
    \right)
\).
If $n \geq p$ and $\mSigma$ is positive-definite, then the elements of $\mA$ have a density over the space of positive-definite matrices, given by
\begin{equation}\label{E:wishart-density}
    f( \mA )
    =
    \frac{ |\mA|^\frac{n-p-1}{2} }
         { 2^\frac{np}{2} 
           |\mSigma|^\frac{n}{2} 
           \Gamma_p \left( \frac{n}{2} \right) }
    \exp \left(
        -
        \frac{1}{2}
        \tr \left(
            \mSigma^{-1} \mA
        \right)
    \right),
\end{equation}
where
\(
    \Gamma_p \left( \cdot \right)
\)
is the multivariate gamma function, computed as
\begin{equation}
    \Gamma_p \left( \frac{n}{2} \right)
    =
    \pi^{p(p-1)/4}
    \prod_{i=1}^p
        \Gamma \left(
            \frac{n + 1 - i}{2}
        \right).
\end{equation}
\end{proposition}

We can now characterize the distributions of the sufficient statistics of a sequence of iid multivariate normal random vectors.

\begin{proposition}
Let 
\(
    \vbX_n
\) 
and 
\(
    \mS_n
\)
be defined as in Proposition~\ref{P:normal-sufficient-stats}.  Then
\(
    \vbX_n
\)
and
\(
    \mS_n
\)
are independent with
\(
    \vbX_n
    \sim
    \Normal \left(
        \mu, \,
        \frac{1}{n}
        \mSigma
    \right)
\)
and
\(
    (n-1)
    \mS_n
    \sim
    \Wishart_p \left(
        n-1, \,
        \mSigma
    \right)
\).
\end{proposition}

White Wishart matrices---those with scale parameter $\mSigma = \sigma^2 \mI_p$
are of particular interest.  We can characterize their distribution in
terms of eigenvalues and eigenvectors.

\begin{proposition}
Suppose that
\(
    \mA
    \sim
    \Wishart_p \left(
        n, \,
        \sigma^2 \mI_p
    \right)
\)
with
\(
    n \geq p
\)
and let
\(
    \mA = n \mO \mL \mO^\trans
\)
be the spectral decomposition of
\(
    \mA,
\)
with
\(
    \mL
    =
    \diag \left(
        l_1,
        l_2,
        \ldots,
        l_p
    \right)
\)
and
\(
    l_1
    >
    l_2
    >
    \cdots
    >
    l_p
    >
    0.
\)
Then $\mO$ and $\mL$ are independent, with $\mO$ Haar-distributed over
the group of $p \times p$ orthogonal matrices and the elements of $\mL$
having density
\begin{equation}\label{E:wishart-eig-density}
    \left(
        \frac{1}{2 \sigma^2 }
    \right)^{np/2}
    \frac{ \pi^{p^2/2} }
         { \Gamma_p \left( \frac{n}{2} \right) 
           \Gamma_p \left( \frac{p}{2} \right) }
    \prod_{i < j}^p
        \left| l_i - l_j \right|
    \prod_{i=1}^p
        l_i^{(n-p-1)/2}
        e^{-\tfrac{l_i}{2 \sigma^2}}
        .
\end{equation}    
\end{proposition}
\noindent
In the random matrix theory literature, with $\sigma^2 = 1$ the eigenvalue density above is sometimes referred to as the Laguerre Orthogonal Ensemble (LOE).

\section{Classical asymptotics}\label{S:multivariate-classical}

In this section we present results about sample covariance matrices when the sample size, $n$, tends to infinity, with the number of dimensions, $p$ a fixed constant.  A straightforward application of the strong law of large numbers gives us the limits of the sample mean and covariance.

\begin{proposition}\label{P:suff-stat-limits}
Let $\vX_1, \vX_2, \ldots, \vX_n$ be a sequence of iid random vectors in $\reals^p$ with
\(
    \E \left[
        \vX_1
    \right]
    =
    \vmu
\)
and
\(
    \E \left[
        \left( \vX_1 - \vmu \right)
        \left( \vX_1 - \vmu \right)^\trans
    \right]
    =
    \mSigma.
\)
Then, as $n \to \infty$,
\[
    \vbX_n
    \define
    \frac{1}{n}
    \sum_{i=1}^n
        \vX_i
    \toas
    \vmu
\]
and
\[
    \mS_n
    \equiv
    \frac{1}{n-1}
    \sum_{i=1}^{n}
        \left( \vX_i - \vbX_n \right)
        \left( \vX_i - \vbX_n \right)^\trans
    \toas
    \mSigma.
\]
\end{proposition}

To simplify matters, for the rest of the section we will mostly work in a setting when the variables have been centered.  In this case, the sample covariance matrix takes the form
\(
    \mS_n = \frac{1}{n} \sum_{i=1}^n \vX_i \vX_i^\trans.
\)
To see that centering the variables does not change the theory in any essential way, we provide the following proposition.

\begin{proposition}
Let $\vX_1, \vX_2, \ldots, \vX_n$ be a sequence of random observations in $\reals^p$ with mean vector $\vmu$ and covariance matrix $\mSigma$.  Let
\(
    \vbX_n = \frac{1}{n} \sum_{i=1}^n \vX_i
\)
and
\(
    \mS_n
    =
    \frac{1}{n-1}
    \sum_{i=1}^n
        \left( \vX_i - \vbX_n \right) \!
        \left( \vX_i - \vbX_n \right)^\trans
\)
be the sample mean and covariance, respectively.  Define the centered
variables
\(
    \vtX_i = \vX_i - \vmu.
\)
Then
\[
    \mS_n
    =
    \frac{1}{n}
    \sum_{i=1}^n
        \vtX_i \vtX_i^\trans
    +
    \OhP\left( \frac{1}{n} \right).
\]
In particular, this implies that
\[
    \sqrt{n}
    \left(
        \mS_n - \mSigma
    \right)
    =
    \sqrt{n}
    \left(
        \frac{1}{n}
        \sum_{i=1}^n
            \vtX_i \vtX_i^\trans
        -
        \mSigma
    \right)
    + 
    \OhP \left( \frac{1}{\sqrt{n}} \right).
\]
\end{proposition}
\begin{proof}
We can write
\begin{align*}
    \mS_n &= \frac{1}{n-1}
             \sum_{i=1}^n \vtX_i \vtX_i^\trans
             +
             \frac{n}{n-1}
             \left(
                \vbX_n - \vmu
             \right)
             \left(
                \vbX_n - \vmu
             \right)^\trans \\
          &= \left(
                \frac{1}{n}
                +
                \frac{1}{n (n-1) }
             \right)
             \sum_{i=1}^n \vtX_i \vtX_i^\trans
             +
             \frac{n}{n-1}
             \left(
                \vbX_n - \vmu
             \right)
             \left(
                \vbX_n - \vmu
             \right)^\trans
\end{align*}
The result follows since
\(
    \vtX_i \vtX_i^\trans = \OhP \left( 1 \right)
\)
and
\(
    \vbX_n - \vmu 
    = 
    \OhP \left( \frac{1}{ \sqrt{n} } \right).
\)

\end{proof}

The next fact follows directly from the multivariate central limit theorem.

\begin{proposition}\label{P:sample-cov-limit}
Suppose that 
\(
    \vX_1, \vX_2, \ldots, \vX_n
\)
is a sequence of iid random vectors in $\reals^p$ with 
\[
    \E \left[
        \vX_1 \vX_1^\trans
    \right]
    =
    \mSigma,
\]
and that for all $1 \leq i,j,i',j' \leq p$ there exists finite
\(
    \Gamma_{iji'j'}
\)
with
\[
    \E \left[
        \left(
            X_{1i} X_{1j}
            -
            \Sigma_{ij}
        \right)
        \left(
            X_{1i'} X_{1j'}
            -
            \Sigma_{i'j'}
        \right)
    \right]
    =
    \Gamma_{iji'j'}.
\]
If
\(
    \mS_n
    =
    \frac{1}{n}
    \sum_{i=1}^n
        \vX_i \vX_i^\trans,
\)
then
\(
    \sqrt{n}
    \vecm \left( 
        \mS_n - \mSigma 
    \right)
    \tod
    \vecm \left( 
        \mG
    \right),
\)
where $\mG$ is a random $p \times p$ symmetric matrix with 
\(
    \vecm \left( \mG \right)
\)
a mean-zero multivariate normal having covariance
\(
    \cov \left(
        G_{ij}, \,
        G_{i'j'}
    \right)
    =
    \Gamma_{iji'j'}.
\)
\end{proposition}

\noindent
In the previous proposition, $\vecm$ is an operator that stacks the columns of
an $n\times p$ matrix to create an $n p$-dimensional vector.

If the elements of $\vX_1$ have vanishing first and third moments (for instance if
the distribution of $\vX_1$ is symmetric about the origin, i.e.
\(
    \vX_1 \eqd - \vX_1
\)),
and if
\(
    \E \left[
        \vX_1 \vX_1^\trans
    \right]
    =
    \diag \left(
        \lambda_1,
        \lambda_2,
        \ldots,
        \lambda_p
    \right),
\)
then $\Gamma_{iji'j'}$ simplifies to
\begin{subequations}
\begin{align}
    \Gamma_{iiii} &= \E \left[ X_{1i}^4 \right] - \lambda_i^2 \quad 
                         &&\text{for $1 \leq i \leq p$}, \\
    \Gamma_{ijij} = \Gamma_{ijji} &= \E \left[ X_{1i}^2 X_{1j}^2 \right]
                         &&\text{for $ 1 \leq i,j \leq p$, $i \neq j$;}
\end{align}
\end{subequations}
all other values of $\Gamma_{iji'j'}$ are $0$.  In particular, this
implies that the elements of $\vecm \left( \mG \right)$ are independent.
If we also have that $\vX_1$ is multivariate normal, then 
\begin{subequations}
\begin{align}
    \Gamma_{iiii} &= 2 \lambda_i^2
                         &&\text{for $1 \leq i \leq p$, and} \\    
    \Gamma_{ijij} = \Gamma_{ijji} &= \lambda_i \lambda_j
                         &&\text{for $ 1 \leq i,j \leq p$, $i \neq j$.}
\end{align}
\end{subequations}


It is inconvenient to study the properties of the sample covariance matrix when the population covariance
\(
    \mSigma
\)
is not diagonal.  By factorizing
\(
    \mSigma
    =
    \mPhi
    \mLambda
    \mPhi^\trans
\)
for orthogonal $\mPhi$ and diagonal $\mLambda$, we can introduce
\(
    \vtX_i
    \define
    \mPhi^\trans \vX_i
\)
to get
\(
    \E \left[
        \vtX_i
        \vtX_i^\trans
    \right]
    =
    \mLambda
\)
and
\(
    \mS_n
    =
    \mPhi
    \left(
        \frac{1}{n}
        \sum_{i=1}^n
            \vtX_i \vtX_i^\trans
    \right)
    \mPhi^\trans.
\)
With this transformation, we can characterize the distribution of $\mS_n$ completely in terms of
\(
    \frac{1}{n}
    \sum_{i=1}^n
        \vtX_i \vtX_i^\trans.
\)


The next result we present is about the sample principal components.  It is motivated by Proposition~\ref{P:sample-cov-limit} and is originally due to Anderson~\cite{anderson1963atp}.

\begin{theorem}\label{T:eigen-random-perturb}
For $n\to \infty$ and $p$ fixed, let $\mS_1, \mS_2, \ldots, \mS_n$ be a sequence of random symmetric $p \times p$ matrices with 
\(
    \sqrt{n} \vecm \left( \mS_n - \mLambda \right)
    \tod
    \vecm \left( \mG \right),
\)
for a deterministic
\(
    \mLambda = \diag\left( \lambda_1, \lambda_2, \ldots, \lambda_p \right)
\)
having $\lambda_1 > \lambda_2 > \cdots > \lambda_p$ and a random symmetric matrix $\mG$.
Let
\(
    \mS_n = \mU_n \mL_n \mU_n^\trans
\)
be the eigendecomposition of $\mS_n$, with
\(
    \mL_n = \diag\left( l_{n,1}, l_{n,2}, \ldots, l_{n,p} \right)
\)
and
\(
    l_{n,1} \geq l_{n,2} \geq \cdots \geq l_{n,p}.
\)
If $\mG = \OhP\left( 1 \right)$, and the signs of $\mU_n$ are chosen so that $U_{n,ii} \geq 0$ for $1 \leq i \leq p$, then the elements of $\mU_n$ and $\mL_n$ converge jointly as
\begin{subequations}
\begin{alignat}{2}
    \sqrt{n}
    \left( U_{n, ii} - 1 \right)
        &\toP 0
                &&\quad \text{for $1 \leq i \leq p$,} \\
    \sqrt{n}
    U_{n, ij}
        &\tod -
              \frac{ G_{ij} }{ \lambda_i - \lambda_j }
                &&\quad \text{for $1 \leq i,j \leq p$ with $i \neq j$, and} \\
    \sqrt{n}
    \left( l_{n,i} - \lambda_i \right)
        &\tod G_{ii}
                &&\quad \text{for $1 \leq i \leq p$.} \qedhere
\end{alignat}
\end{subequations}
\end{theorem}
\noindent
More generally, Anderson treats the case when the $\lambda_i$ are not all unique.  The key ingredient to Anderson's proof is a perturbation lemma, which we state and prove below.
\begin{lemma}\label{L:eigen-perturb}
For 
\(
    n \to \infty
\)
and fixed
\(
    p
\)
let
\(
    \mS_1, \mS_2, \ldots, \mS_n \in \reals^{p\times p}
\) 
be a sequence of symmetric matrices of the form
\[
    \mS_n 
    = 
    \mLambda
    +
    \frac{1}{\sqrt{n}}
    \mH_n
    +
    \oh\left( \frac{1}{\sqrt{n}} \right),
\]
where
\(
    \mLambda
    =
    \diag \left(
        \lambda_1,
        \lambda_2,
        \ldots,
        \lambda_p
    \right)
\)
with 
\(
    \lambda_1 > \lambda_2 > \cdots > \lambda_p
\) 
and
\(
    \mH_n = \Oh\left( 1 \right).
\)
Let $\mS_n = \mU_n \mL_n \mU_n^\trans$ be the eigendecomposition of $\mS_n$, with
\(
    \mL_n
    =
    \diag \left(
        l_{n,1}, l_{n,2}, \ldots, l_{n,p}
    \right).
\)
Further suppose that $U_{n,ii} \geq 0$ for $1 \leq i \leq p$ and
\(
    l_{n,1} > l_{n,2} > \cdots > l_{n,p}.
\)
Then for all $1 \leq i,j \leq p$ and $i \neq j$ we have
\begin{subequations}
\begin{align}
    U_{n,ii} 
        &= 1 
           + 
           \oh \left( 
               \frac{ 1 }{ \sqrt{n} }
           \right), \\
    U_{n,ij}
        &= -
           \frac{ 1 }{ \sqrt{n} }
           \frac{ H_{n,ij} }{ \lambda_i - \lambda_j }
           +
           \oh \left(
               \frac{ 1 }{ \sqrt{n} }
           \right), \quad \text{and} \\
    l_{n,i}
        &= \lambda_i
           + 
           \frac{ H_{n,ii} }{ \sqrt{n} }
           +
           \oh \left(
               \frac{ 1 }{ \sqrt{n} }
           \right).
\end{align}
\end{subequations}
\end{lemma}
\begin{proof}
Define $p\times p$ matrices $\mE_n$, $\mF_n$, and $\mDelta_n$ so that
\begin{align}
    \mE_n
        &=
        \diag \left(
            U_{n,11},
            U_{n,22},
            \ldots,
            U_{n,pp}
        \right), \\
    \mF_n 
        &= 
        \sqrt{n} \left( \mU_n - \mE_n \right), \\
    \mDelta_{n}
        &=
        \sqrt{n} \left( \mL_n - \mLambda \right), \\
\intertext{giving}
    \mU_n
        &= \mE_n + \frac{1}{\sqrt{n}} \mF_n, \quad \text{and} \notag \\
    \mL_n
        &= \mLambda + \frac{1}{\sqrt{n}} \mDelta_{n}. \notag
\end{align}
We have that 
\begin{align}
    \mS_n 
    &= \mLambda 
       + \frac{1}{\sqrt{n}} \mH_n 
       + \oh \left( \frac{1}{\sqrt{n}}\right) \notag \\
    &= \mU_n \mL_n \mU_n^\trans \notag \\
    &= \mE_n \mLambda \mE_n^\trans
       + 
       \frac{1}{\sqrt{n}} \left(
           \mE_n \mDelta_n \mE_n^\trans
           +
           \mF_n \mLambda \mE_n^\trans
           +
           \mE_n \mLambda \mF_n^\trans
       \right)
       +
       \frac{1}{n}
       \mM_n \label{L:eigen-perturb:E:eigen-perturb}
\end{align}
where the elements of $\mM_n$ are sums of $\Oh\left( p \right)$ terms, with each term a product of elements taken from $\mE_n$, $\mF_n$, $\mLambda$, and $\mDelta_n$.  Also,
\begin{align}
    \mI_p
    &= \mU_n \mU_n^\trans \notag \\
    &= \mE_n \mE_n^\trans
       + 
       \frac{1}{\sqrt{n}} \left(
           \mF_n \mE_n^\trans
           +
           \mE_n \mF_n^\trans
       \right)
       + 
       \frac{1}{n}
       \mW_n, \label{L:eigen-perturb:E:orthog-perturb}
\end{align}
where $\mW_n = \mF_n \mF_n^\trans$.
From \eqref{L:eigen-perturb:E:orthog-perturb} we see that for $1 \leq i,j \leq p$ and $i \neq j$ we must have
\begin{subequations}
\begin{align}
    1 &= E_{n,ii}^2 
         + 
         \frac{1}{n} W_{n,ii}, 
         \quad \text{and}
         \label{L:eigen-perturb:E:orthog-perturb-1} \\
    0 &= E_{n,ii} F_{n,ji} 
         + 
         F_{n,ij} E_{n,jj} 
         + 
         \frac{1}{\sqrt{n}} W_{n,ij}.
         \label{L:eigen-perturb:E:orthog-perturb-2}
\end{align}
\end{subequations}
Substituting
\(
    E_{n,ii}^2 = 1 - \frac{1}{n} W_{n,ii}
\)
into equation~\eqref{L:eigen-perturb:E:eigen-perturb}, we get
\begin{subequations}
\begin{align}
    H_{n,ii} 
        &= E_{n,ii} \Delta_{n,ii} E_{n,ii}
           + 
           \frac{1}{\sqrt{n}} \left(
                M_{n,ii} - \lambda_i W_{n,ii}
           \right)
           +
           \oh \left( 1 \right), \quad \text{and} 
           \label{L:eigen-perturb:E:eigen-perturb-1} \\
    H_{n,ij}
        &= \lambda_j E_{n,jj} F_{n,ij}
           +
           \lambda_i F_{n,ji} E_{n,ii} 
           +
           \frac{1}{\sqrt{n}} M_{n,ij}
           +
           \oh \left( 1 \right).
           \label{L:eigen-perturb:E:eigen-perturb-2} 
\end{align}
\end{subequations}
Equations~\eqref{L:eigen-perturb:E:orthog-perturb-1}--\eqref{L:eigen-perturb:E:eigen-perturb-2} admit the solution
\begin{subequations}
\begin{align}
    E_{n,ii} 
        &= 1 + \oh\left(\frac{1}{\sqrt{n}}\right), \\
    F_{n,ij}
        &= -
           \frac{ H_{n,ij} }
                { \lambda_i - \lambda_j }
           +
           \oh \left( 1 \right), \quad \text{and} \\
    \Delta_{n,ii}
        &= H_{n,ii} + \oh\left( 1 \right). 
\end{align}
\end{subequations}
This completes the proof.
\end{proof}

An application of the results of this section is the following theorem, which describes the behavior of principal component analysis for large $n$ and fixed $p$.

\begin{theorem}
Let $\vX_1, \vX_2, \ldots, \vX_n$ be a sequence of iid 
\(
    \Normal \left(
        \vmu, \,
        \mSigma
    \right)
\)
random vectors in $\reals^p$, with sample mean
\(
    \vbX_n
\)
and sample covariance
\(
    \mS_n.
\)
Let $\mSigma = \mPhi \mLambda \mPhi^\trans$ be the eigendecomposition
of $\mSigma$, with
\(
    \mLambda = \diag \left(
        \lambda_1,
        \lambda_2,
        \ldots,
        \lambda_p
    \right)
\)
and
\(
    \lambda_1 > \lambda_2 > \cdots > \lambda_p > 0
\).
Similarly, let $\mS_n = \mU_n \mL_n \mU_n^\trans$ be the eigendecomposition of $\mS_n$, likewise with
\(
    \mL_n = \diag \left(
        l_{n,1},
        l_{n,2},
        \ldots,
        l_{n,p}
    \right)
\),
\(
    l_{n,1} > l_{n,2} > \cdots > l_{n,p}
\),
and signs chosen so that
\(
    \left( \mPhi^\trans \mU_n \right)_{ii} \geq 0
\)
for $1 \leq i \leq p$.  Then
\begin{enumerate}[(i)]
    \item $l_{n,i} \toas \lambda_i$ for
        $1 \leq i \leq p$, and $\mU_n \toas \Phi$ .
    \item 
        After appropriate cenering and scaling, 
	$\{ l_{n,i} \}_{i=1}^p$ and $\mU_n$
        converge jointly in distribution and their limits
        are independent.	
	For all $1 \leq i \leq p$
        \[
            \sqrt{n} \left( l_{n,i} - \lambda_i \right) 
            \tod 
            \Normal\left( 0, \, 2 \lambda_i^2 \right),
        \]
        and
        \[
            \sqrt{n} \left( \mPhi^\trans \mU_n - \mI_p \right) \tod \mF,
        \]
        where $\mF$ is a skew-symmetric matrix with elements above the 
	diagonal independent of each other and distributed as 
        \[
            F_{ij}
            \sim
            \Normal \left(
                0, \,
                \frac{\lambda_i \lambda_j}
                     {\left( \lambda_i - \lambda_j \right)^2}
            \right),
            \quad
            \text{for all $1 \leq i < j \leq p$.}
        \]
\end{enumerate}
\end{theorem}
\begin{proof}
Part (i) is a restatement of Proposition~\ref{P:suff-stat-limits}.  Part (ii) follows from Proposition~\ref{P:sample-cov-limit} and Theorem~\ref{L:eigen-perturb}.
\end{proof}


\section{Modern asymptotics}\label{S:multivariate-modern}

We now present some results about sample covariance matrices when both the sample size, $n$, and the dimensionality, $p$, go to infinity.  Specifically, most of these results suppose that $n \to \infty$, $p \to \infty$, and 
\(
    \frac{n}{p} \to \gamma
\)
for a fixed constant
\(
    \gamma \in \left( 0, \infty \right).
\)
There is no widely-accepted name for $\gamma$, but we will sometimes adopt the terminology of Mar\v{c}enko and Pastur \cite{marcenko1967des} and call it the \emph{concentration}.

Most of the random matrix theory literature concerning sample covariance matrices is focused on eigenvalues.  Given a sequence of sample covariance matrices $\mS_1, \mS_2, \ldots, \mS_n$, with $\mS_n \in \reals^{p\times p}$ and $p = p(n)$ these results generally come in one of two forms.  If we label the eigenvalues of $\mS_n$ as
\(
    l_{n,1}, l_{n,2}, \ldots, l_{n,p},
\)
with $l_{n,1} \geq l_{n,2} \geq \cdots \geq l_{n,p}$, then we can define
a random measure
\begin{equation}
    F^{S_n} = \frac{1}{p} \sum_{i=1}^p \delta_{l_{n,i}}.
\end{equation}
This measure represents a random draw from the set of eigenvalues of $\mS_n$ that puts equal weight on each eigenvalue.  It is called the \emph{spectral measure} of $\mS_n$.  Results about $F^{S_n}$ are generally called results about the ``bulk'' of the spectrum.  

The second major class of results is concerned with the behavior of the
extreme eigenvalues $l_{n,1}$ and $l_{n,p}$. Results of this type are called
``edge'' results.

\subsection{The bulk of the spectrum}

To work in a setting where the dimensionality $p$, grows with the sample size, $n$, we introduce a triangular array of sample vectors.  The sample covariance matrix $\mS_n$ is of dimension $p\times p$ and is formed from row $n$ of a triangular array of independent random vectors,  
$\vX_{n,1}, \vX_{n,2}, \ldots, \vX_{n,n}$.  Specifically,
\(
    \mS_n
    =
    \frac{1}{n}
    \sum_{i=1}^n \vX_{n,i} \vX_{n,i}^\trans.
\)
We let $\mX_n$ be the $n \times p$ matrix
\(
    \mX_n
    =
    \left(
    \begin{matrix}
        \vX_{n,1} &
        \vX_{n,2} &
        \cdots &
        \vX_{n,n}
    \end{matrix}
    \right)^\trans,
\)
so that 
\(
    \mS_n
    =
    \frac{1}{n}
    \mX_n^\trans \mX_n.
\)
Most asymptotic results about sample covariance matrices are expressed in terms of $\mX_n$ rather than $\mS_n$.  For example, the next theorem about the spectral measure of a large covariance matrix is stated this way.

\begin{theorem}\label{T:mp-limit}
Let $\mX_1, \mX_2, \ldots, \mX_n$ be a sequence of random matrices of increasing dimension as $n \to \infty$, so that $\mX_n \in \reals^{n\times p}$ and $p = p(n)$.
Define $\mS_n = \frac{1}{n} \mX_n^\trans \mX_n$.  If the elements of $\mX_n$ are iid with $\E| X_{n,11} - \E X_{n,11} |^2 = 1$ and $\frac{n}{p} \to \gamma > 0$, then the empirical spectral measure $F^{\mS_n}$ almost surely converges in distribution to a deterministic probability measure.  This measure, denoted
$\FMP_\gamma$, is called the Mar\v{c}enko-Pastur Law.  For $\gamma \geq 1$ it
has density
\begin{equation}
    \fMP_\gamma(x)
    =
    \frac{\gamma}{2 \pi x}
    \sqrt{(x - a_\gamma)(b_\gamma - x)},
    \quad
    a_\gamma \leq x \leq b_\gamma,
\end{equation}
where
\(
    a_\gamma = \left( 1 - \frac{1}{\sqrt{\gamma}} \right)^2
\)
and
\(
    b_\gamma = \left( 1 + \frac{1}{\sqrt{\gamma}} \right)^2.
\)
When $\gamma < 1$, there is an additional point-mass of $(1 - \gamma)$ at the origin.
\end{theorem}

\noindent
Figure~\ref{F:mp-law} shows the density $\fMP_\gamma(x)$ for different values
of $\gamma$.  The reason for choosing the name ``concentration'' to refer to $\gamma$ becomes apparent in that for larger values of $\gamma$, $\FMP_\gamma$ becomes more and more concentrated around its mean.

The limiting behavior of the empirical spectral measure of a sample covariance matrix was originally studied by Mar\v{c}enko and Pastur~\cite{marcenko1967des} in 1967.  Since then, several papers have refined these results, including Grenander and Silverstein~\cite{grenander1977san}, Wachter~\cite{wachter1978slr},
 Jonsson~\cite{jonsson1982slt}, Yin and Krishnaiah~\cite{yin1983lte},
Yin~\cite{yin1986lsd}, Silverstein and Bai~\cite{silverstein1995ede}, and
Silverstein~\cite{silverstein1995sce}.  These papers either proceed via a combinatorial argument involving the moments of the matrix elements, or else they employ a tool called the Stieltjes transform.  Theorem~\ref{T:mp-limit} is a simplified version of Silverstein and Bai's main result, which more generally considers complex-valued random variables and allows the columns of $\mX_n$ to have heterogeneous variances.

\begin{figure}
    \centering
    \includegraphics{mp-law}
    \caption{
        \captiontitle{The Mar\v{c}enko-Pastur law}
        Density, $\fMP_\gamma(x)$, plotted against quantile, $x$,
        for concentration $\gamma = 0.25$, $1$, and $4$.  Concentration
        is equal to the number of samples per dimension. For $\gamma < 1$,
        there is an addition point-mass of size $(1 - \gamma)$ at $x = 0$.
    }\label{F:mp-law}
\end{figure}

The meaning of the phrase ``$F^{\mS_n}$ converges almost surely in distribution to $\FMP_\gamma$'' is that for all $x$ which are continuity points of $\FMP_\gamma$,
\begin{equation}\label{E:spectral-measure-limit-point}
    F^{\mS_n} \left(
        x
    \right)
    \toas
    \FMP_\gamma \left(
        x
    \right).
\end{equation}
Equivalently, Theorem~\ref{T:mp-limit} can be stated as a strong law of large numbers.

\begin{corollary}[Wishart LLN]\label{C:wishart-lln}
    Let $\mX_n$ and $\{ l_{n,i} \}_{i=1}^{p}$ be as in 
    Theorem~\ref{T:mp-limit}.  Let $g : \reals \to \reals$ be any
    continuous bounded function.  Then
    \begin{equation}\label{E:wishart-functional-limit}
        \frac{1}{p}
        \sum_{i=1}^p
            g \left( l_{n,i} \right)
        \toas
        \int
            g(x)
            \,
            d\FMP_\gamma ( x ).
    \end{equation}
\end{corollary}

Concerning convergence rates for the quantities in Theorem~\ref{T:mp-limit},  Bai et al. \cite{bai2003crs} study the total variation distance between $F^{\mS_n}$ and $\FMP_\gamma$.  Under suitable conditions on $X_{n,11}$ and $\gamma$, they show that
\(
    \| F^{\mS_n} - \FMP_\gamma \|_\text{TV}
    =
    \OhP\left( n^{-2/5} \right)
\)
and that with probability one
\(
    \| F^{\mS_n} - \FMP_\gamma \|_\text{TV}
    =
    \Oh\left( n^{-2/5 + \varepsilon} \right)
\)
for any $\varepsilon > 0$.
Guionnet and Zeitouni \cite{guionnet2000csm} give concentration of measure results.  If $X_{n,11}$ satisfies a Poincar\'e inequality and $g$ is Lipschitz, they show that for $\delta$ large enough,
\begin{equation}\label{E:wishart-ldp}
    -
    \frac{1}{n^2}
    \log \Prob\left\{ 
        \left|
            \int g(x) \, dF^{\mS_n}(x) - \int g(x) \, d\FMP_\gamma(x)
        \right|
        >
        \delta
    \right\}
    =
    \Oh\left(
        \delta^2
    \right),
\end{equation}
with an explicit bound on the error. If one is willing to assume that the elements of $\mX_n$ are Gaussian, then Hiai and Petz~\cite{hiai1998edw} give an exact value for the quantity in \eqref{E:wishart-ldp}.  Guionnet~\cite{guionnetlds} gives a survey of other large deviations results.  

It is interesting to look at the scaled behavior in equation~\eqref{E:wishart-functional-limit} when the quantities are scaled by $p$.  Indeed one can prove a Central Limit Theorem (CLT) for functionals of eigenvalues.

\begin{theorem}[Wishart CLT]\label{T:wishart-clt}
    Let $\mX_1, \mX_2, \ldots, \mX_n$ be a sequence of random $n\times p$ 
    matrices with $p = p(n)$.  Assume that $\mX_n$ has iid elements and that 
    $\E\left[ X_{n,11} \right] = 0$, $\E\left[ X_{n,11}^2 \right] = 1$,
    and $\E\left[ X_{n,11}^4 \right] < \infty$.  
    Define $\mS_n = \frac{1}{n} \mX_n^\trans \mX_n$ and let $F^{\mS_n}$ be its 
    spectral measure.  If $n \to \infty$, $\frac{n}{p} \to \gamma$ and $g_1, 
    g_2, \ldots, g_k$ 
    are real-valued functions analytic on the support of $\FMP_\gamma$, then 
    the sequence of random vectors
    \begin{multline*}
        p
        \cdot
        \Big(
	    \int g_1(x) \, dF^{\mS_n}(x)
	    -
            \int g_1(x) \, d\FMP(x), \\
	    \int g_2(x) \, dF^{\mS_n}(x)
	    -
	    \int g_2(x) \, d\FMP(x),
	    \ldots, \\
            \int g_k(x) \, dF^{\mS_n}(x)
	    -
            \int g_k(x) \, d\FMP(x)
	\Big)
    \end{multline*}
    is tight.
    Moreover, if $\E \left[ X_{n,11}^4 \right] = 3$, then the sequence 
    converges in distribution to a multivariate normal with mean $\vmu$ and 
    covariance $\mSigma$, where 
    \begin{equation}\label{E:wishart-clt-mean}
        \mu_i
	    =
	    \frac{ g_i(a_\gamma) + g_i(b_\gamma) }
                 { 4 }
	    -
            \frac{1}{2 \pi}
	    \int_{a_\gamma}^{b_\gamma}
	        \frac{ g_i(x) }
	             { \sqrt{(x-a_\gamma)(b_\gamma-x)} }
                \,
                dx
    \end{equation}
    and
    \begin{equation}\label{E:wishart-clt-cov}
    	\Sigma_{ij}
    	=
    	-
    	\frac{1}{2 \pi^2}
    	\iint
    	    \frac{g_i(z_1) g_j(z_2)}
    	         {\left( m(z_1) - m(z_2) \right)^2}
    	    \frac{d}{dz_1} m(z_1)
    	    \frac{d}{dz_2} m(z_2)
    	    dz_1 dz_2.
    \end{equation}
    The integrals in~\eqref{E:wishart-clt-cov} are contour integrals enclosing the 
    support of $\FMP_\gamma$, and
    \begin{equation}
        m(z)
        =
	    \frac{-(z + 1 - \gamma^{-1}) + \sqrt{(z-a_\gamma)(z-b_\gamma)}}{2 z},
    \end{equation}
    with the square root defined to have positive imaginary part when
    $\Im z > 0$.
\end{theorem}

\noindent
The case when $\E\left[ X_{n,11}^4 \right] = 3$ is of particular interest
because it arises when $X_{n,11} \sim \Normal\left( 0, 1 \right)$.

This theorem was proved by Bai and Silverstein~\cite{bai2004clt}, and can
be considered a generalization of the work by Johnsson~\cite{jonsson1982slt}.
In computing the variance integral~\eqref{E:wishart-clt-cov}, it is useful to know that $m$ satisfies
the identities 
\[
    m(\bar z) = \overline{m (z) },
\]
and
\[
    z = - \frac{1}{m(z)} + \frac{\gamma^{-1}}{m(z) + 1}.
\]
Bai and Silverstein show how to compute the limiting means and variances
for $g(x) = \log x$ and $g(x) = x^r$.  They also derive a similar CLT when
the elements of $\mX_n$ are correlated.  Pastur and Lytova~\cite{pastur2008clt} have recently relaxed some of the assumptions made by Bai and Silverstein.


\subsection{The edges of the spectrum}

We now turn our attention to the extreme eigenvalues of a sample covariance
matrix.  It seems plausible that if $F^{\mS_n} \tod \FMP_\gamma$, then the
extreme eigenvalues of $\mS_n$ should converge to the edges of the support
of $\FMP_\gamma$.  Indeed, under suitable assumptions, this is exactly what
happens.  For the largest eigenvalue, work on this problem started with Geman~\cite{geman1980ltn}, and his assumptions were further weakened by Jonsson~\cite{jonsson1983ole} and Silverstein~\cite{silverstein1984ole}.
Yin et al.~\cite{yin1988lle} prove the result under the weakest possible
conditions~\cite{bai1988nle}.

\begin{theorem}\label{T:max-wishart-eig-limit}
    Let $\mX_1, \mX_2, \ldots, \mX_n$ be a sequence of random matrices of
    increasing dimension, with $\mX_n$ of size $n \times p$, $p = p(n)$, 
    $n \to \infty$, and
    $\frac{n}{p} \to \gamma \in (0,\infty)$.  Let $\mS_n = \frac{1}{n} \mX_n^\trans \mX_n$ and 
    denote its eigenvalues by 
    $l_{n,1} \geq l_{n,2} \geq \cdots \geq l_{n,p}$.  If the 
    elements of $\mX_n$ are 
    iid with $\E X_{n,11} = 0$, $\E X_{n,11}^2 = 1$, and
    $\E X_{n,11}^4 < \infty$, then
    \[
        l_{n,1} \toas \left( 1 + \gamma^{-1/2} \right)^2.
    \]
\end{theorem}

For the smallest eigenvalue, the first work was by Silverstein~\cite{silverstein1985sel}, who gives a result when 
$X_{n,11} \sim \Normal\left( 0,\, 1 \right)$.  Bai and Yin~\cite{bai1993lse} proved a theorem that mirrors Theorem~\ref{T:max-wishart-eig-limit}.

\begin{theorem}\label{T:min-wishart-eig-limit}
    Let $\mX_n$, $n$, $p$, and $\{ l_{n,i} \}_{i=1}^p$ be as in
    Theorem~\ref{T:max-wishart-eig-limit}.  If $\E X_{n,11}^4 < \infty$ and
    $\gamma \geq 1$, then
    \[
        l_{n,p} \toas \left( 1 - \gamma^{-1/2} \right)^2.
    \]
    With the same moment assumption on $X_{n,11}$, if $0 < \gamma < 1$, then
    \[
        l_{n,p-n+1} \toas \left( 1 - \gamma^{-1/2} \right)^2.
    \]
\end{theorem}

\noindent
For the case when the elements of $\mX_n$ are correlated, Bai and Silverstein~\cite{bai1998neo} give a general result that subsumes Theorems~\ref{T:max-wishart-eig-limit}~and~\ref{T:min-wishart-eig-limit}.

After appropriate centering and scaling, the largest eigenvalue of a white
Wishart matrix converges weakly to a random variable with known distribution.
Johansson~\cite{johansson2000sfa} proved this statement and identified the
limiting distribution for complex white Wishart matrices.  Johnstone~\cite{johnstone2001dle} later provided an analogous result for real matrices, which we state below.  

\begin{theorem}\label{T:tw-limit-largest}
    Let $\mX_1, \mX_2, \ldots, \mX_n$ be a sequence of random matrices of
    increasing dimension, with $\mX_n \in \reals^{n\times p}$, $p = p(n)$,
    and $n \to \infty$ with $\frac{n}{p} \to \gamma \in (0, \infty)$.  Define
    $\mS_n = \frac{1}{n} \mX_n^\trans \mX_n$ and label its eigenvalues
    \(
        l_{n,1} \geq l_{n,2} \geq \cdots \geq l_{n,p}.
    \)
    If the elements of $\mX_n$ are iid with
    \(
        X_{n,11} \sim \Normal\left( 0, \, 1 \right),
    \)
    then
    \[
        \frac{l_{n,1} - \mu_{n,p}}{\sigma_{n,p}}
        \tod
        W_1
        \sim
        \FTW_1,
    \]
    where
    \begin{align*}
        \mu_{n,p} 
            &=
            \frac{1}{n}
            \left(
                \sqrt{n - 1/2}
                +
                \sqrt{p - 1/2}
            \right), \\
        \sigma_{n,p}
            &= 
            \frac{1}{n}
            \left(
                \sqrt{n - 1/2}
                +
                \sqrt{p - 1/2}
            \right)
            \left(
                \frac{1}{\sqrt{n - 1/2}}
                +
                \frac{1}{\sqrt{p - 1/2}}
            \right)^{1/3},
    \end{align*}
    and $\FTW_1$ is the Tracy-Widom law of order 1.
\end{theorem}

\noindent
El Karoui~\cite{elkaroui2003lew} extended this result to apply when $\gamma = 0$ or $\gamma = \infty$.  With appropriate modifications to $\mu_{n,p}$ and $\sigma_{n,p}$, he later gave a convergence rate of order $(n \wedge p)^{2/3}$  for complex-valued data \cite{elkaroui2006mpt}.  Ma~\cite{ma2008atw} gave the analogous result for real-valued data.  For correlated complex normals, El Karoui~\cite{elkaroui2007twl} derived a more general version of Theorem~\ref{T:tw-limit-largest}.  

The Tracy-Widom distribution, which appears in Theorem~\ref{T:tw-limit-largest}, was  established to be the limiting distribution (after appropriate scaling) of the maximum eigenvalue from an $n \times n$ symmetric matrix with independent entries distributed as $\Normal\left( 0, \, 2 \right)$ along the main diagonal and $\Normal\left( 0, \, 1 \right)$ otherwise \cite{tracy1994lsd} \cite{tracy1996oas}.  To describte $\FTW_1$, let $q(x)$ solve the Painlev\'e II equation
\[
    q''(x) = x q(x) + 2 q^3(x),
\]
with boundary condition $q(x) \sim \Ai(x)$ as $x \to \infty$ and $\Ai(x)$ the Airy function.  Then it follows that
\[
    \FTW_1(x)
    =
    \exp \left\{
        -
        \frac{1}{2}
        \int_s^\infty
            q(x)
            +
            (x-s) q^2 (x)
            \,
            dx
    \right\}.
\]
Hastings and McLeod~\cite{hastings1980bvp} study the tail behavior of $q(x)$.  Using their analysis, one can show (see, e.g. \cite{perry2009mre}) that for $s \to -\infty$, 
\[
    \FTW_1(s)
    \sim
    \exp \left(
        -\frac{|s|^3}{24}
    \right),
\]
while for $s \to \infty$,
\[
    1 - \FTW_1(s)
    \sim
    \frac{ s^{-3/4} }
         { 4 \sqrt{ \pi } }
    \exp \left(
        -
        \frac{2}{3}
        s^{3/2}
    \right).
\]
The density of $\FTW_1$ is shown in Figure~\ref{F:tw-density}.

\begin{figure}
    \centering
    \includegraphics{tw-law}
    \caption{
        \captiontitle{The Tracy-Widom law}
        Limiting density of the largest eigenvalue from a white Wishart
        matrix after appropriate centering and scaling.
    }
    \label{F:tw-density}
\end{figure}

A result like Theorem~\ref{T:tw-limit-largest} holds true for the smallest eigenvalue.  We define the \emph{Reflected Tracy-Widom Law} to have distribution function $\GTW_1(s) = 1 - \FTW_1(-s)$.  Then we have 

\begin{theorem}\label{T:tw-limit-smallest}
    With the same assumptions as in Theorem~\ref{T:tw-limit-largest}, if
    $\gamma \in (1,\infty)$ then
    \[
        \frac{l_{n,p} - \mu_{n,p}^-}{\sigma_{n,p}^-}
        \tod
        W_1
        \sim
        \GTW_1,
    \]
    where
    \begin{align*}
        \mu_{n,p}^-
            &=
            \frac{1}{n}
            \left(
                \sqrt{n - 1/2}
                -
                \sqrt{p - 1/2}
            \right), \\
        \sigma_{n,p}^-
            &= 
            \frac{1}{n}
            \left(
                \sqrt{n - 1/2}
                -
                \sqrt{p - 1/2}
            \right)
            \left(
                \frac{1}{\sqrt{p - 1/2}}
                -
                \frac{1}{\sqrt{n - 1/2}}
            \right)^{1/3}.
    \end{align*}
    If $\gamma \in (0, 1)$, then
    \[
        \frac{l_{n,p-n+1} - \mu_{p,n}^-}{\sigma_{p,n}^-}
        \tod
        W_1
        \sim
        \GTW_1.
    \]
\end{theorem}

\noindent
We get the result for $\gamma \in (0,1)$ by reversing the role of $n$ and $p$.
Baker et al~\cite{baker1998rme} proved the result for complex data.
Paul~\cite{paul2006dse} extended the result to real data when $\gamma \to \infty$.  Ma~\cite{ma2008atw} gives convergence rates.  In practice,
$\log l_{n,p}$ converges in distribution faster than $l_{n,p}$.  Ma recommends appropriate centering and scaling constants for $\log l_{n,p}$ to converge in distribution to a $\GTW$ random variable at rate $(n \wedge p)^{2/3}$.

Theorem~\ref{T:tw-limit-smallest} does not apply when $\frac{n}{p} \to 1$.  Edelman~\cite{edelman1988ecn} derived the limiting distribution of the smallest eigenvalue when $n = p$.  It is not known if his result holds more generally when $\frac{n}{p} \to 1$.

\begin{theorem}\label{T:edelman-limit-smallest}
    Let $\mX_n$ and $\{ l_{n,i} \}_{i=1}^p$ be as in 
    Theorem~\ref{T:tw-limit-largest}.  If $p(n) = n$, then for $t \geq 0$,
    \[
        \Prob\left\{
            n \, l_{n,p} \leq t
        \right\}
        \to
        \int_0^t
            \frac{1 + \sqrt{x}}{\sqrt{x}}
            e^{-(x/2 + \sqrt{x})}
            \,
            dx.
    \]
\end{theorem}

In addition to the extreme eigenvalues, it is possible to study the joint distribution of top or bottom $k$ sample eigenvalues for fixed $k$ as
$n \to \infty$.  In light of Theorem~\ref{T:mp-limit}, for fixed $k$ we must have that the top (respectively, bottom) sample eigenvalues converge almost surely to the same limit.  Furthermore, Soshnikov~\cite{soshnikov2002nud} showed that applying the centering and scaling from Theorem~\ref{T:tw-limit-largest} to the top $k$ sample eigenvalues gives a specific limiting distribution.

It is natural to ask if the limiting eigenvalue distributions are specific to Wishart matrices, or if they apply to non-Gaussian data as well.  There is compelling evidence that the Tracy-Widom law is universal.   Soshnikov~\cite{soshnikov2002nud} extended Theorem~\ref{T:tw-limit-largest} to more general $\mX_n$ under the assumption that $X_{n,11}$ is sub-Gaussian and $n-p = \Oh(p^{1/3})$.  P\'ech\'e~\cite{peche2008url} later removed the restriction on $n-p$.  Tao and Vu~\cite{tao2009rmd} showed that Theorem~\ref{T:edelman-limit-smallest} applies for general $X_{n,11}$ with $\E X_{n,11} = 0$ and $\E X_{n,11}^2 = 1$.


\subsection{Eigenvectors}

Relatively little attention has been focused on the eigenvectors of sample covariance matrices.  While many results are known, as of yet there is no complete characterization of the eigenvectors from a general sample covariance matrix.  Most of the difficulty in tackling the problem is that it is hard to describe convergence properties of $\mU_n$, the $p\times p$ matrix of eigenvectors, when $n$ and $p$ go to infinity.  The individual $p^2$ elements of $\mU_n$ do not converge in any meaningful way, so the challenge is to come up with relevant macroscopic characteristics of $\mU_n$.

Silverstein~\cite{silverstein1979reg} was perhaps the first to study the eigenvectors of large-dimensional sample covariance matrices.  He hypothesized that for sample covariance matrices of increasing dimension, the eigenvector matrix becomes more and more ``Haar-like''.  A random matrix $\mU \in \reals^{p\times p}$ is said to be Haar-distributed over the orthogonal group if for every fixed $p \times p$ orthogonal matrix $\mO$, the rotated matrix $\mO \mU$ has the same distribution as $\mU$.  That is, $\mO \mU \eqd \mU$.  Silverstein's conjecture was that as $n \to \infty$, $\mU_n$ behaves more and more like a Haar-distributed matrix.  The next theorem displays one aspect of Haar-like behavior.

To state the theorem, we need to define the extension of a scalar function
$g : \reals \mapsto \reals$ to symmetric matrix arguments.  If 
$\mS = \mU \mL \mU^\trans$ is the eigendecomposition of the symmetric
matrix $\mS \in \reals^{p \times p}$, with 
$\mL = \diag(l_1, l_2, \ldots, l_p)$, then we define
\[
    g\left( \mS \right)
        =
            \mU
            \Big(
                \diag\big(
                    g(l_1), g(l_2), \ldots, g(l_p)
                \big)
            \Big)
            \mU^\trans.
\]
With this notion, we can state Silverstein's result.

\begin{theorem}\label{T:evec-functional-limit}
    Let $\mX_1, \mX_2, \ldots, \mX_n$ be a sequence of random matrices of
    increasing dimension, with $\mX_n \in \reals^{n \times p}$, $p = p(n)$,
    and $\mX_n$ having iid elements with 
    \(
        \E X_{n,11} = 0,
    \)
    \(
        \E X_{n,11}^2 = 1,
    \)
    and
    \(
        \E X_{n,11}^4 < \infty.
    \)
    Define $\mS_n = \frac{1}{n} \mX_n^\trans \mX_n$.  Let
    $\va_1, \va_2, \ldots, \va_n$ be a sequence of nonrandom unit vectors with 
    $\va_n$ in $\reals^p$ and let $g : \reals \to \reals$ be a continuous 
    bounded function.  If $n \to \infty$ and
    $\frac{n}{p} \to \gamma \in (0,\infty)$, then
    \[
        \va_n^\trans g\left( \mS_n \right) \va_n
        \toas
        \int
            g(x)
            \,
            d\FMP_\gamma(x).
    \]
\end{theorem}

\noindent
Silverstein~\cite{silverstein1979reg} proves the result for convergence in probability and a specific class of $\mX_n$.  Bai et al.~\cite{bai2007ael} strengthen the result to a larger class of $\mX_n$ and proves almost-sure convergence.  They also consider dependence in $\mX_n$.

It may not be immediately obvious how Theorem~\ref{T:evec-functional-limit}
is related to eigenvectors.  If $\mS_n = \mU_n \mL_n \mU_n$ is the
eigendecomposition of $\mS_n$, with 
\(
    \mL_n
    = 
    \diag\left(
        l_{n,1}, l_{n,2}, \ldots, l_{n,p}
    \right),
\)
then
\(
    g(\mL_n)
    =
    \diag \big(
        g(l_{n,1}), g(l_{n,2}), \ldots, g(l_{n,p})
    \big)
\)
and
\(
    g(\mS_n)
    =
    \mU_n
    g(\mL_n)
    \mU_n^\trans.
\)
We let $\vb_n = \mU_n \va_n$.  Then,
\begin{equation}\label{E:evector-functional}
    \va_n^\trans g(\mS_n) \va_n
    =
    \sum_{i=1}^p
        b_{n,i}^2
        \,
        g( l_{n,i} ).
\end{equation}
If $\mU_n$ is Haar-distributed, then $\vb_n$ will be distributed uniformly over the unit sphere in $\reals^p$, and the average in \eqref{E:evector-functional} will put about weight $\frac{1}{p}$ on each eigenvalue.  If $\mU_n$ puts bias in any particular direction then the average will put extra weight on particular eigenvalues.

Silverstein investigated second-order behavior of eigenvectors in \cite{silverstein1981dbe}, \cite{silverstein1984slt}, \cite{silverstein1989eld}, and \cite{silverstein1990wcr}.  He demonstrated that certain second-order behavior of $\mU_n$ depends in a crucial way on the fourth moment of $X_{n,11}$.  This greatly restricts the class of $\mX_n$ for with the eigenvectors of $\mS_n$ are Haar-like.

\begin{theorem}\label{T:evec-functional-scaled-limit}
    Let $\mX_n$, $\mS_n$, and $\va_n$ be as in 
    Theorem~\ref{T:evec-functional-limit}. Suppose also that
    $\E X_{n,11}^4 = 3$.  Let $g_1, g_2, \ldots, g_k$
    be real-valued functions analytic on the support of $\FMP_\gamma$.  
    Then, the random vector
    \begin{multline*}
        \sqrt{p} \cdot
        \left(
            \va_n^\trans g_1( \mS_n ) \va_n
            -
            \int g_1(x) \, d\FMP_\gamma(x), \right. \\
            \va_n^\trans g_2( \mS_n ) \va_n
            -
            \int g_2(x) \, d\FMP_\gamma(x),
            \ldots, \\ \left.
            \va_n^\trans g_k( \mS_n ) \va_n
            -
            \int g_k(x) \, d\FMP_\gamma(x)
        \right)
    \end{multline*}
    converges in distribution to a mean-zero multivariate normal with
    covariance between the $i$th and $j$th components equal to
    \[
        \int
            g_i(x) g_j(x) \, d\FMP_\gamma(x)
        -
        \int
            g_i(x) \, d\FMP_\gamma(x)
        \cdot
        \int
            g_j(x) \, d\FMP_\gamma(x).
    \]
\end{theorem}

\noindent
Bai et al.~\cite{bai2007ael} give a similar result for complex-valued and correlated $\mX_n$.  Silverstein~\cite{silverstein1989eld} showed that if $g_1(x) = x$ and $g_2(x) = x^2$, then the condition $\E X_{n,11}^4 = 3$ is necessary for the random vector in Theorem~\ref{T:evec-functional-scaled-limit} to converge in distribution for all $\va_n$.  However, for the specific choice of
\(
    \va_n
    =
    \left(
        \frac{1}{\sqrt{p}},
        \frac{1}{\sqrt{p}},
        \ldots,        
        \frac{1}{\sqrt{p}}        
    \right),
\)
he later showed that the conclusions of Theorem~\ref{T:evec-functional-scaled-limit} hold more generally when $X_{n,11}$ is symmetric and $\E X_{n,11}^4 < \infty$ \cite{silverstein1990wcr}.  

perry-thesis.tex
%\documentclass[12pt,openright]{report}
\documentclass[12pt,twoside,openright]{report}
\pdfoutput=1 % ensure that arXiv uses pdflatex
% note that the documentclass can take other option such as
% twoside - for double sided printing
% openright - if double side  chapters always start on odd pages
% openany - if double side chapters start on the next page even or odd
% 12pt can be replaced by 11pt


\usepackage{suthesis-2e}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{enumerate}
\usepackage{epsfig}
\usepackage{perry-thesis-macros}

\DeclareGraphicsExtensions{.png,.pdf}

\newcommand{\dirsep}{/}
\newcommand{\dirprefix}{}
\newcommand{\thesischapter}[1]{\dirprefix #1\dirsep #1}
\newcommand{\includechapter}[1]{
    \graphicspath{{\dirprefix #1\dirsep plots\dirsep}}
    \include{\thesischapter{#1}}
}
\renewcommand{\bibname}{References}

    \title{Cross-Validation for Unsupervised Learning}
    \author{Patrick O. Perry}
    \submitdate{September 2009}
    \copyrightyear{2009}
    \dept{Statistics}
    \principaladviser{Art B. Owen}
    \firstreader{Iain M. Johnstone}
    \secondreader{Jonathan E. Taylor}

\includeonly{
    acknowledgment,
    {\thesischapter{introduction}},
    {\thesischapter{multivariate}},
    {\thesischapter{lowrank}},
    {\thesischapter{intrinsic-rank}},
    {\thesischapter{cv-unsupervised}},
    {\thesischapter{bcv-theory}},
    {\thesischapter{projections}},
    {\thesischapter{weighted-sums}},
    bibliography
}


\begin{document}

\beforepreface

\prefacesection{Abstract}
\input{abstract}

\include{acknowledgment}

\figurespagefalse
\tablespagefalse
\afterpreface

\includechapter{introduction}
\includechapter{multivariate}
\includechapter{lowrank}
\includechapter{intrinsic-rank}
\includechapter{cv-unsupervised}
\includechapter{bcv-theory}

\appendix

\includechapter{projections}
\includechapter{weighted-sums}

\include{bibliography}

\end{document}

projections.tex

\chapter{Properties of random projections}\label{A:projections}

We use this section to present some results about random projection
matrices.  We call a symmetric $p \times p$ matrix $\mP$ a projection 
if its eigenvalues are in the set $\{ 0, 1 \}$.  Any projection matrix of rank
$k \leq p$ can be decomposed as $\mP = \mV \mV^\trans$ for some $\mV$
satisfying $\mV^\trans \mV = \mI_k$.  We call the set 
\begin{equation}
    \stiefel_{k}\left( \reals^p \right)
        =
        \left\{
            \mV \in \reals^{p \times k}
            :
            \mV^\trans \mV = \mI_k
        \right\}
        \subseteq
        \reals^{p \times k}
\end{equation}
the \emph{rank-$k$ Stiefel manifold of $\reals^p$}.  It is the set
of orthonormal $k$-frames in $\reals^p$.  Similarly, we call 
the set
\begin{equation}
    \grassmann_k\left( \reals^p \right)
        =
        \left\{
            \mV \mV^\trans
            :
            \mV \in \stiefel_k \left( \reals^p \right)
        \right\}
        \subseteq
        \reals^{p \times p}
\end{equation}
the \emph{rank-$k$ Grassmannian of $\reals^p$}; this is the set of
rank $k$ projection matrices.  If
\(
    \left(
    \begin{matrix}
        \mV & \mbV
    \end{matrix}
    \right)
\)
is an $p \times p$ Haar-distributed orthogonal matrix and $\mV$ is
$p \times k$, then we say that $\mV$ is uniformly distributed
over $\stiefel_k( \reals^p )$ and that $\mV \mV^\trans$ is uniformly
distributed over $\grassmann_k(\reals^p)$.


\section{Uniformly distributed orthonormal $k$-frames}

We first present some results about a matrix $\mV$ distributed uniformly
over $\stiefel_k(\reals^p)$.  We denote this distribution by
\(
    \mV 
        \sim
            \Unif \big(
                \stiefel_k(\reals^p)
            \big).
\)
In the special case of $k = 1$, the distribution is equivalent to drawing
a random vector uniformly from the unit sphere in $\reals^p$.  We denote 
this distribution by
\(
    \vV
        \sim
            \Unif \big(
                \sphere^{p-1}
            \big).
\)


\subsection{Generating random elements}

The easiest way to generate a random element of $\stiefel_k(\reals^p)$ is
to let $\mZ$ be a $p \times k$ matrix of iid $\Normal( 0, \, 1)$
random variables, take the $QR$ decomposition $\mZ = \mQ \mR$, and let
$\mV$ be equal to the first $k$ columns of $\mQ$.  In practice, there is
a bias in the way standard $QR$ implementations choose the signs of the
columns of $\mQ$.  To get around this, we recommend using 
Algorithm~\ref{A:random-steifel}, below.

\begin{algorithm}
    \caption{\label{A:random-steifel}Generate a random orthonormal $k$-frame}
    \begin{enumerate}
        \item Draw $\mZ$, a random $p \times k$ matrix whose elements are 
              iid $\Normal( 0, \, 1 )$ random variables.
              
        \item Compute $\mZ = \mQ \mR$, the $QR$-decomposition of $\mZ$.
              Set $\mQ_1$ to be the $p \times k$ matrix containing the
              first $k$ columns of $\mQ$.

        \item Draw $\mS$, a random $k \times k$ diagonal matrix with iid 
              diagonal entries such that 
              \(
                  \Prob\left\{ S_{11} = -1 \right\}
                  =
                  \Prob\left\{ S_{11} = +1 \right\}
                  =
                  \frac{1}{2}.
              \)
        \item Return $\mV = \mQ_1 \mS$.
    \end{enumerate}
\end{algorithm}

\noindent 
This algorithm has a time complexity of $\Oh\left( p k^2 \right)$. Diaconis
and Shahshahani~\cite{diaconis1987sag} present an alternative approach called
the subgroup algorithm which can be used to generate $\mV$ as a product of 
$k$ Householder reflections.  Their algorithm has time complexity 
$\Oh\left( p k \right)$.  Mezzadri~\cite{mezzadri2007grm} gives a simple 
description of the subgroup algorithm.


\subsection{Mixed moments}

Since we can flip the sign of any row or column of $\mV$ and not change
its distribution, the mixed moments of the elements of $\mV$ vanish unless 
the number of elements from any row or column is even (counting multiplicity).  For example, $\E \left[ V_{11}^2 V_{21} \right] = 0$ since we can flip the sign of the second row of $\mV$ to get
\[
    V_{11}^2 V_{21} 
    \eqd
    V_{11}^2 (- V_{21})
    =
    -V_{11}^2 V_{21}.
\]
This argument does not apply to $V_{11} V_{12} V_{22} V_{21}$ or
$V_{11}^2 V_{22}^2$.


In the special case when $k = 1$, the vector
\(
    \left(
        V_{11}^2, V_{21}^2, \ldots, V_{p1}^2
    \right)
\)
is distributed as 
\(
    \Dirichlet\left( \frac{1}{2}, \frac{1}{2}, \ldots, \frac{1}{2} \right).
\)
This follows from the fact that if $Y_1, Y_2, \ldots, Y_p$ are independently distributed Gamma random variables and $Y_i$ has shape
$a_i$ and scale $s$, then with $S = \sum_{i=1}^p Y_i$, the vector
\(
    \left(
        \frac{Y_1}{S},
        \frac{Y_2}{S},
        \ldots,
        \frac{Y_p}{S}
    \right)
\)
is distributed $\Dirichlet\left( a_1, a_2, \ldots, a_p \right)$.  Using
the standard formulas for Dirichlet variances and covariances, we get
the mixed moments up to fourth order.  They are summarized in the next
lemma.

\begin{lemma}\label{L:stiefel-1-moments}
    If $\mV \sim \Unif\big( \stiefel_1(\reals^p) \big)$, then
    \begin{subequations}
    \begin{align}
        \E \left[ V_{11}^2 \right] 
            &= \frac{1}{p}, \\
        \E \left[ V_{11} V_{21} \right] 
            &= 0, \\
        \E \left[ V_{11}^4 \right] 
            &= \frac{3}{p \, (p + 2)}, \\
        \E \left[ V_{11}^2 V_{21}^2 \right] 
            &= \frac{1}{p \, (p + 2)}.
    \end{align}
    \end{subequations}
    The odd mixed moments are all equal to zero.
\end{lemma}

Using Theorem~4 from Diaconis and Shahshahani~\cite{diaconis1994erm},
which gives the moments of the traces of Haar-distributed orthogonal
matrices, we can compute the mixed moments of $\mV$ for more general $k$.
Meckes~\cite{meckes2006ivs} gives an alternative derivation of these
results.

\begin{lemma}\label{L:stiefel-k-moments}
    If $\mV \sim \Unif\big( \stiefel_k(\reals^p) \big)$ and $k > 1$, then the
    nonzero mixed moments of the elements $\mV$ up to fourth order
    are defined by
    \begin{subequations}
    \begin{align}
        \E \left[ V_{11}^2 \right] 
            &= 
                \frac{1}{ p }, \\
        \E \left[ V_{11}^4 \right] 
            &=
                \frac{3}{ p \, (p+2) }, \\
        \E \left[ V_{11}^2 V_{21}^2 \right] 
            &= 
                \frac{1}{ p \, (p+2) }, \\
        \E \left[ V_{11}^2 V_{22}^2 \right]
            &= 
                \frac{p+1}{ p \, (p-1) (p+2) }, \\
        \E \left[ V_{11} V_{12} V_{22} V_{21} \right] 
            &= 
                \frac{ -1 }{ p \, (p-1) (p+2) }.
    \end{align}
    \end{subequations}
\end{lemma}
\begin{proof}
    The first three equations follow directly from the previous lemma.
    We can get the other moments from the moments of $\mO$, a Haar-distibuted
    $p \times p$ orthogonal matrix. For the fourth equation, we use that 
    \[
        \E \left[ \tr( \mO ) \right]^4
        =
        \sum_{r,s,t,u} \E \left[ O_{rr} O_{ss} O_{tt} O_{uu} \right].
    \]
    Only the terms with even powers of $O_{ii}$ are nonzero.  Thus, we have
    \begin{align*}
        \E \left[ \tr( \mO ) \right]^4
        &=   \binom{p}{1} \E \left[ O_{11}^4 \right]
           + \binom{p}{2} \binom{4}{2} \E \left[ O_{11}^2 O_{22}^2 \right] \\
        &=   p \, \E \left[ O_{11}^4 \right]
           + 3p \, (p-1) \, \E \left[ O_{11}^2 O_{22}^2 \right].
    \end{align*}
    Theorem~4 of Diaconis and Shahshahani \cite{diaconis1994erm} gives that 
    $\E \left[ \tr(\mO) \right]^4 = 3$.  Combined with
    Lemma~\ref{L:stiefel-1-moments}, we get that
    \begin{align*}
        \E \left[ O_{11}^2 O_{22}^2 \right]
        &= \frac{1}{3p \, (p-1)} 
            \left\{
                \E \left[ \tr(\mO) \right]^4
                - p \, \E \left[ O_{11}^4 \right]
            \right\} \\
        &= \frac{1}{3p \, (p-1)} 
           \left\{
                3 - p \cdot \frac{3}{ p \, (p+1) }
           \right\} \\
        &= \frac{p+1}{ p \, (p-1) (p+2) }.
    \end{align*}
    
    For the last equation, we use $\E\left[ \tr(\mO^4) \right]$.  We have that 
    \[
        (\mO^4)_{ij} = \sum_{r,s,t} O_{ir} O_{rs} O_{st} O_{tj}.
    \]
    We would like to compute $\E\left[ (\mO^4)_{11} \right]$.  Note that
    unless $r=s=t=1$, there are only three situations when
    $\E \left[ O_{1r} O_{rs} O_{st} O_{t1} \right] \neq 0$.   Two of them are 
    demonstrated visually by the configurations
    \[
        \begin{matrix}
            \phantom{0} &
            \begin{matrix} 
                1 & \phantom{\cdots} & s \phantom{\cdots}
            \end{matrix} \\
            \begin{matrix}
                1 \\
                \phantom{\vdots}  \\
                s \\
                \phantom{\vdots}
            \end{matrix}&
            \left(
            \begin{matrix}
                O_{1r} & \cdots & O_{rs} & \cdots \\
                \vdots  & \ddots & \vdots  & \\
                O_{t1} & \cdots & O_{st} & \cdots \\
                \vdots  &        & \vdots  &
            \end{matrix}
            \right) \\
            \phantom{0} &
            r=1, s=t, s \neq 1
        \end{matrix}
        \quad\quad\text{and}\quad\quad
        \begin{matrix}
            \phantom{0} &
            \begin{matrix} 
                1 & \phantom{\cdots} & r \phantom{\cdots}
            \end{matrix} \\
            \begin{matrix}
                1 \\
                \phantom{\vdots}  \\
                r \\
                \phantom{\vdots}
            \end{matrix}&
            \left(
            \begin{matrix}
                O_{t1} & \cdots & O_{1r} & \cdots \\
                \vdots  & \ddots & \vdots  & \\
                O_{st} & \cdots & O_{rs} & \cdots \\
                \vdots  &        & \vdots  &
            \end{matrix}
            \right) \\
            \phantom{0} &
            t=1, r=s, r \neq 1
        \end{matrix}.
    \]
    The other nonzero term is when $s=1$, $r=t$, and $r\neq1$, so
    that $O_{1r} O_{rs} O_{st} O_{t1} = O_{1r}^2 O_{r1}^2$.
    In all other configurations there is a row or a column that only
    contains one of $\{ O_{1r}, \, O_{rs}, \, O_{st}, \, O_{t1} \}$.  Since 
    we can multiply a row or a column of $\mO$ by $-1$ and not change the
    distribution of $\mO$, for this choice of $r$, $s$, and $t$ we must have
    $O_{1r} O_{rs} O_{st} O_{t1} \overset{d}{=} -O_{1r} O_{rs} O_{st} O_{t1}$.
    This in turn implies that 
    $\E\left[O_{1r} O_{rs} O_{st} O_{t1} \right] = 0$.
    With these combinatorics in mind, we have that
    \begin{align*}
        \E \! \left[ (\mO^4)_{11} \right]
        \!&=\!
            \sum_{s \neq 1} \E \left[ O_{11} O_{1s} O_{ss} O_{s1} \right]
            +
            \sum_{r \neq 1} \E \left[ O_{1r} O_{rr} O_{r1} O_{11} \right]
            +
            \sum_{r \neq 1} \E \left[ O_{1r}^2 O_{r1}^2 \right]
            + 
            \E \left[ O_{11}^4 \right] \\
        &=
            2(p-1) \E \left[ O_{11} O_{12} O_{22} O_{21} \right]
            + (p-1) \E \left[ O_{12}^2 O_{21}^2 \right]
            + \E \left[ O_{11}^4 \right] \\
        &=
            2(p-1) \E \left[ O_{11} O_{12} O_{22} O_{21} \right]
            + (p-1) \E \left[ O_{11}^2 O_{22}^2 \right]
            + \E \left[ O_{11}^4 \right]
    \end{align*}
    Again applying Theorem~4 of \cite{diaconis1994erm}, we have that 
    $\E \left[ \tr (\mO^4) \right] = 1$.  Combined with 
    Lemma~\ref{L:stiefel-1-moments}, we
    have
    \begin{align*}
        \E \left[ O_{11} O_{12} O_{22} O_{21} \right]
        &= 
        \frac{1}{2(p-1)}
        \left\{
            \frac{1}{p} \E \left[ \tr (\mO^4) \right]
            - \E\left[ O_{11}^4 \right]
            - (p-1) \E \left[ O_{11}^2 O_{22}^2 \right]
        \right\} \\
        &=
        \frac{1}{2(p-1)}
        \left\{
            \frac{1}{p}
            - \frac{3}{ p \, (p+2) }
            - (p-1) \frac{p+1}{ p \, (p-1) (p+2) }
        \right\} \\
        &= 
        \frac{-1}{ p \, (p-1) (p+2) }. \qedhere
    \end{align*} 
\end{proof}


\section{Uniformly distributed projections}

When $\mP \in \reals^{p \times p}$ is chosen uniformly over the set of rank-$k$ $p \times p$ projection matrices, we say
\(
    \mP
        \sim
            \Unif \big(
                \grassmann_k(\reals^p)
            \big).
\)
With the results of the previous section, we can derive the moments of
random projection matrices.


\begin{lemma}\label{L:grassman-moments-1}
    If $\mP \sim \Unif \big( \grassmann_k(\reals^p) \big)$, then
    \begin{align}
        \E \left[ \mP \right] &= \frac{k}{p} \mI_p.
    \end{align}
\end{lemma}
\begin{proof}
    Write $\mP = \mV \mV^\trans$, where 
    \(
        \mV \sim \Unif \big( \stiefel_k( \reals^p ) \big)
    \).
    For $1 \leq i,j \leq p$ we have
    \[
        P_{ij}
            =
                \sum_{r=1}^k V_{ir} V_{jr},
    \]
    so
    \[
        \E \left[ P_{ij} \right] 
        =
        \begin{cases}
            k \, \E \left[ V_{11}^2 \right], &\text{when $i = j$,} \\
            k \, \E \left[ V_{11} V_{21} \right], &\text{otherwise.}
        \end{cases}
    \]
    The result now follows from Lemma~\ref{L:stiefel-1-moments}.
\end{proof}

\begin{lemma}\label{L:grassman-moments-2}
    Let $\mP \sim \Unif \big( \grassmann_k(\reals^p) \big)$.  If
    $1 \leq i,j,i',j' \leq p$, then
    \begin{equation}
        \begin{split}
            \cov \big[ P_{ij}, P_{i'j'} \big]
                &=
                    \frac{1}{(p-1)(p+2)}
                    \left(
                        \frac{k}{p}
                    \right)
                    \left(
                        1 - \frac{k}{p}
                    \right) \\
                &\qquad\cdot
                    \Big(
                          p \, \delta_{(i,j ) = (i',j')}
                        + p \, \delta_{(i,j ) = (j',i')}
                        - 2 \, \delta_{(i,i') = (j ,j')}
                    \Big).
        \end{split}
    \end{equation}
    This gives us that aside from the obvious symmetry ($P_{ij} = P_{ji}$), 
    the off-diagonal elements of $\mP$ are uncorrelated with each other and 
    with the diagonal elements.
\end{lemma}
\begin{proof}
    We need to perform six computations.  As before, we use the representation
    $\mP = \mV \mV^\trans$, where 
    \(
        \mV \sim \Unif\big( \stiefel_k(\reals^p) \big).
    \)
    We have
    \begin{align*}
        \E \left[ P_{11}^2 \right]
        &= \E \bigg[ \Big( 
                         \sum_{i=1}^k V_{1i}^2
                     \Big)^2
              \bigg] \\
        &= \E \bigg[
                  \sum_{i=1}^k V_{1i}^4
                  + \sum_{i\neq j} V_{1i}^2 V_{1j}^2
              \bigg] \\
        &= k \cdot \frac{3}{ p \, (p+2) } 
           + k \, (k-1) \cdot \frac{1}{ n \, (p+2) } \\
        &= \frac{ k \, (k+2) }{ p \,(p+2) } \\
        &= \frac{2}{ p+2}
           \left(\frac{k}{p}\right) \left( 1 - \frac{k}{p} \right)
           + \left( \frac{k}{p} \right)^2,
    \end{align*}
    which gives us that 
    \[
        \var \left[ P_{11} \right]
        = \frac{2}{ p+2 }
           \left(\frac{k}{p}\right) \left( 1 - \frac{k}{p} \right).
    \]
    Next,
    \begin{align*}
        \E \left[ P_{12}^2 \right]
        &= \E \bigg[ \Big( 
                         \sum_{i=1}^k V_{1i} V_{2i}
                     \Big)^2
              \bigg] \\
        &= \E \bigg[ 
                  \sum_{i=1}^k V_{1i}^2 V_{2i}^2
                  + \sum_{i\neq j} 
                        V_{1i} V_{2i} V_{1j} V_{2j}
              \bigg] \\
        &= k \cdot \frac{ 1 }{ p \, (p+2) }
           + k \, (k-1) \cdot \frac{-1}{ p \, (p-1) (p+2) } \\
        &= \frac{p}{ (p-1) (p+2) } 
           \left(\frac{k}{p}\right) \left( 1 - \frac{k}{p} \right),
    \end{align*}
    so that
    \[
        \var \left[ P_{12} \right]
        = \frac{p}{ (p-1) (p+2) } 
           \left(\frac{k}{p}\right) \left( 1 - \frac{k}{p} \right).
    \]
    Also,
    \begin{align*}
        \E \left[ P_{11} P_{22} \right]
        &= \E \bigg[
                \Big( \sum_{i=1}^k V_{1i}^2 \Big)
                \Big( \sum_{j=1}^k V_{2j}^2 \Big)
              \bigg] \\
        &= \E \bigg[
                  \sum_{i=1}^k V_{1i}^2 V_{2i}^2
                  + \sum_{i=1}^k
                    \sum_{j\neq i} V_{1i}^2 V_{2j}^2
              \bigg] \\
        &= k \cdot \frac{1}{ p \, (p+2) }
           + k \, (k-1) \cdot \frac{ p+1 }{ p \, (p-1) (p+2) } \\
        &= \frac{k}{ p \, (p+2) }
           \left( 1 + (p+1) \frac{k-1}{p-1} \right) \\
        &= \frac{-2}{(p-1)(p+2)}
           \left(\frac{k}{p}\right) \left( 1 - \frac{k}{p} \right)
           + \left( \frac{k}{p} \right)^2,
    \end{align*}
    so that
    \[
        \cov \left[ P_{11}, P_{22} \right]
        = \frac{-2}{ (p-1) (p+2) }
           \left(\frac{k}{p}\right) \left( 1 - \frac{k}{p} \right).
    \]
    Since $\mP$ is symmetric, we have
    \[
        \cov \left[ P_{12}, P_{21} \right]
        = \var \left[ P_{12} \right].
    \]
    The other covariances are all zero.  This is because
    \begin{align*}
        \E \left[ P_{11} P_{12} \right]
        &= \E \bigg[ 
                  \sum_{i,j} V_{1i}^2 \, V_{1j}  V_{2j} 
              \bigg], \\
        \E \left[ P_{12} P_{23} \right]
        &= \E \bigg[ 
                \sum_{i,j} V_{1i} V_{2i} V_{2j} V_{3j}
              \bigg], \\
    \intertext{and}
        \E \left[ P_{12} P_{34} \right]
        &= \E \bigg[ 
                \sum_{i,j} V_{1i} V_{2i} V_{3j} V_{4j}
              \bigg].
    \end{align*}
    Each term in these sums has an element that appears only once in a row.
    Thus, the expectations are all $0$.
\end{proof}


\section{Applications}

We now present two applications of the results in this section.

\subsection{Projections of orthonormal $k$-frames}

Suppose we have $\mU \in \reals^{p \times k}$, an orthonormal $k$-frame, and we randomly project the columns of $\mU$ into $\reals^q$, with $q < p$.  If we denote the projection matrix by $\mV^\trans$ and set $\mtU =\mV^\trans \mU$, it is natural to ask how close $\mtU$ is to being an orthonormal $k$-frame. We can prove the following:

\begin{proposition}\label{P:project-kframe}
    Suppose $\mU \in \reals^{p \times k}$ satisfies $\mU^\trans \mU = \mI_k$.
    Let $\mV \sim \Unif\big( \stiefel_q(\reals^p) \big)$ with 
    $k \leq q \leq p$ and set
    \(
        \mtU = \sqrt{p/q} \mV^\trans \mU.
    \)
    Then there exists a decomposition 
    $\mtU = \mtU_0 + \frac{1}{\sqrt{q}} \mtU_1$ such that
    \(
        \mtU_0^\trans \mtU_0 = \mI_k
    \)
    and
    \begin{equation}
        \E \| \mtU_1 \|_\Frob^2 
            \leq
                \frac{1}{2} \,
                k \, (k + 1)
                \left( \frac{q}{p} \right)^2
                \left( 1 - \frac{q}{p} \right).
    \end{equation}
    In particular, this implies that
    \(
        \E \| \mtU_1 \|_\Frob^2 
            \leq \frac{2}{27} \, k \, (k+1).
    \)
\end{proposition}

\noindent
The main ingredients of the proof are a perturbation lemma and a result about $\mtU^\trans \mtU$, stated below.

\begin{lemma}\label{L:project-kframe-dot-products}
    Suppose $\mU \in \reals^{p \times k}$ satisfies $\mU^\trans \mU = \mI_k$.
    Let $\mV \sim \Unif\big( \stiefel_q(\reals^p) \big)$ with 
    $k \leq q \leq p$
    and set
    \(
        \mtU = \sqrt{p/q} \mV^\trans \mU.
    \)
    Then 
    \(
        \E \left[ \mtU^\trans \mtU \right] = \mI_k
    \)
    and
    \begin{subequations}
    \begin{equation}
        \mE \Big\| 
            \sqrt{q} \, \big(  \mtU^\trans \mtU - \mI_k \big) 
        \Big\|_\Frob^2
                \leq
                k \, (k+1)
                \left( \frac{q}{p} \right)^2                
                \left( 1 - \frac{q}{p} \right),
    \end{equation}
    with a matching lower bound of
    \begin{equation}
        \mE \Big\| 
            \sqrt{q} \, \big(  \mtU^\trans \mtU - \mI_k \big) 
        \Big\|_\Frob^2
            \geq
            k \, (k+1)
            \left( \frac{q}{p} \right)^2
            \left( 1 - \frac{q}{p} \right)
            \left(
            \frac{p}{p+2}
            \right).
    \end{equation}
    \end{subequations}
\end{lemma}
\begin{proof}
    Set $\mP = \mV \mV^\trans$ so that
    \(
        \mtU^\trans \mtU
        =
        \frac{p}{q} \, 
        \mU^\trans \mP \mU.
    \)
    Now,
    \[
        \mE \left[ \mtU^\trans \mtU \right]
            =
                \frac{p}{q} \,
                \mU^\trans
                \E \left[ \mP \right]
                \mU
            =
            \mI_k.
    \]
    Also, since $\mO^\trans \mP \mO \eqd \mP$ for any $p \times p$ orthogonal
    matrix $\mO$, we must that $\mU^\trans \mP \mU$ has the same distribution
    as the upper $k \times k$ submatrix of $\mP$.  This implies that
    \begin{align*}
        \mE \Big\| 
            \mtU^\trans \mtU - \mI_k
        \Big\|_\Frob^2
            &= \sum_{i=1}^k \sum_{j=1}^k
                     \var \left[ \mP_{ij} \right] \\
            &= \frac{q}{p} \left( 1 - \frac{q}{p} \right)
               \bigg\{ 
                    k \cdot \frac{2}{p + 2} 
                    +
                    k \, (k-1)
                    \cdot
                    \frac{p}{ (p-1) (p+2) }
               \bigg\} \\
            &= 
               \frac{p \, k \, (k+1) }{ (p-1) (p+2) }
               \left( \frac{q}{p} \right) \left( 1 - \frac{q}{p} \right)
               \left(
                    1
                    -
                    \frac{2}{ p \, (k+1) }
               \right).
    \end{align*}
    The lower and upper bounds follow.
\end{proof}

The next ingredient is a perturbation theorem due to Mirsky, which we take 
from Stewart~\cite{stewart1990pts} and state as a lemma.

\begin{lemma}[Mirsky]\label{L:mirsky}
    If $\mA$ and $\mA + \mE$ are in $\reals^{n \times p}$, then
    \begin{equation}
        \sum_{i=1}^{n \wedge p} 
            \big(
                \sigma_i(\mA + \mE) - \sigma_i( \mA ) 
            \big)^2 
        \leq \| \mE \|_\Frob^2,
    \end{equation}
    where $\sigma_i(\cdot)$ denotes the $i$th singular value.
\end{lemma}

We can now proceed to the rest of the proof.
\begin{proof}[Proof of Proposition~\ref{P:project-kframe}]
    We have that $\mtU^\trans \mtU = \mI_k + \mE$, 
    where $\E \| \sqrt{q} \mE \|_\Frob^2 \leq C$ and $C$ is
    given in Lemma~\ref{L:project-kframe-dot-products}.
    We can apply Lemma~\ref{L:mirsky} to get
    \[
        \sum_{i=1}^k 
            \big(
                \sigma_i(\mtU^\trans \mtU) - 1
            \big)^2 
        \leq \| \mE \|_\Frob^2.
    \]
    Setting $\epsilon_i = \sigma_i(\mtU^\trans \mtU) - 1$, we have 
    $\sigma_i( \mtU ) = \sqrt{1 + \epsilon_i}$.  Note that 
    \(
        \E \left[ q \sum_i \epsilon_i^2 \right]
        \leq \E \| \sqrt{q} \, \mE \|_\Frob^2
        \leq C.
    \)
    Let $\mR$ and $\mS$ be $p \times k$ and $k \times k$ matrices
    with orthonormal columns such that
    \[
        \mtU = \mR (\mI_k + \mDelta) \mS^\trans
    \]
    is the SVD of $\mtU$, where 
    \(
        \mDelta
            = 
                \diag( 
                    \Delta_1, \Delta_2, \ldots, \Delta_k
                ),
    \)
    and $\Delta_i = \sqrt{ 1 + \epsilon_i } - 1.$  Set
    \(
        \mtU_0 = \mR \mS^\trans
    \)
    and
    \(
        \mtU_1 = \sqrt{q} \, \mR \mDelta \mS^\trans.
    \)
    Then,
    \[
        \mtU_0^\trans \mtU_0 
                = \mS \mR^\trans \mR \mS^\trans
                = \mS \mS^\trans
                = \mI_k
    \]
    and 
    \[
        \E \| \mtU_1 \|_\Frob^2 
            = \sum_{i=1}^k \E \left[ q \, \Delta_i^2 \right]
            \leq \frac{1}{2} \sum_{i=1}^k \E \left[ q \, \epsilon_i^2 \right]
            \leq \frac{C}{2},
    \]
    where we have used that
    \(
        \sqrt{1 + \epsilon_i} 
            \geq 
                1 + \frac{1}{2} \epsilon_i - \frac{1}{2} \epsilon_i^2.
    \)
\end{proof}


\subsection{A probabilistic interpretation of the Frobenius norm}
\label{S:probabilistic-frob}

As another application of the results in this section, we give a probabilistic
representation of the Frobenius norm of a matrix.  It is commonly known that 
for any 
$n \times p$ matrix $\mA$, 
\begin{equation}
    \sup_{ \substack{
                \| \vx \|_2 = 1, \\
                \| \vy \|_2 = 1 } }
        \big( \vx^\trans \mA \, \vy \big)^2
        =
        \| \mA \|_2^2,
\end{equation}
where $\| \cdot \|_2$ is the spectral norm, equal largest singular value
(see, e.g.~\cite{golub1996mc}).  The function
\(
    f( \vx, \vy ) = \vx^\trans \mA \, \vy
\)
is a general bilinear form on $\reals^n \times \reals^p$.  The square of the
spectral norm of $\mA$ gives the maximum value of $\big( f(\vx, \vy) \big)^2$
when $\vx$ and $\vy$ are both unit vectors.
It turns out that the Frobenius norm of $\mA$ is related to the
\emph{average} value of $\big( f(\vX, \vY) \big)^2$ when $\vX$ and $\vY$ are
random unit vectors.

\begin{proposition}\label{P:probabilistic-frob}
    If $\mA \in \reals^{n \times p}$, then
    \begin{equation}
        \int\limits_{ \substack{
                          \| \vx \|_2 = 1, \\
                          \| \vy \|_2 = 1 } }
            \!\!\!\!\!
            \big(
                \vx^\trans \mA \, \vy
            \big)^2
            \, d\vx \, d\vy
        =
        \frac{1}{np} \| \mA \|_F^2.
    \end{equation}
\end{proposition}
\begin{proof}
    There are two steps to the proof.  First, we show that if 
    \(
        \vX \sim \Unif\big( \sphere^{n-1} \big)
    \)
    and $\va$ is arbitrary, then
    \[
        \E \left[ \vX^\trans \va \right]^2 = \frac{1}{n} \| \va \|_2^2.
    \]
    Next, we show that if $\vY \sim \Unif\big( \sphere^{p-1} \big)$, then
    \[
        \E \| \mA \, \vY \|_2^2 = \frac{1}{p} \| \mA \|_\Frob^2.
    \]
    The result follows from these two facts.
    To see the first part, since $\vX$ is orthogonally invariant we have
    \[
        \vX^\trans \va
            \eqd 
                \vX^\trans \left( \| \va \|_2 \, \ve_1 \right)
            = \| \va \|_2 \, \vX_1.
    \]
    To see the second part, let $\mA = \mU \mSigma \mV^\trans$ be the
    SVD of $\mA$ and write
    \[
        \| \mA \, \vY \|_2^2
            =
                \| \mSigma \mV^\trans \vY \|_2^2
            \eqd
                \| \mSigma \vY \|_2^2
            =
                \sum_{i=1}^{n \wedge p} \sigma_i^2( \mA ) \, Y_i^2.
        \qedhere
    \]
\end{proof}

weighted-sums.tex
\chapter{Limit theorems for weighted sums}\label{A:weighted-sums}

In this appendix we state and prove some limit theorems for weighted sums
of iid random variables.  First, we give a weak law of large numbers (WLLN):

\begin{proposition}[Weighted WLLN]\label{P:weighted-wlln}
    Let $X_{n,1}, X_{n,2}, \ldots, X_{n,n}$ be a triangular array of random
    variables, iid across each row, with $\E X_{n,1} = \mu$ and $\E 
    X_{n,1}^2$  uniformly bounded in $n$.  Also, let $W_{n,1}, W_{n,2}, \ldots, 
    W_{n,n}$ be another triangular array of random variables independent of 
    the $X_{n,i}$ (but not necessarily of each other).  Define 
    \(
        \bW_n = \frac{1}{n} \sum_{i=1}^n W_{n,i}.
    \) 
    If $\bW_n \toP \bW$ and 
    $\E \left[ \frac{1}{n} \sum_{i=1}^n W_{n,i}^2 \right]$ is uniformly 
    bounded in $n$, then
    \begin{equation}
        \frac{1}{n} \sum_{i=1}^n W_{n,i} \, X_{n,i} \toP \bW \mu.
    \end{equation}
\end{proposition}

We do not give a proof of Proposition~\ref{P:weighted-wlln}, but instead derive a strong law of large numbers (SLLN) below.  The proof of the weak law is similar.  Here is the strong law:

\begin{proposition}[Weighted SLLN]\label{P:weighted-slln}
    Let $X_{n,1}, X_{n,2}, \ldots, X_{n,n}$ be a triangular array of random
    variables, iid across each row, with $\E X_{n,1} = \mu$ and 
    $\E X_{n,1}^4$  uniformly bounded in $n$.
    Also, let $W_{n,1}, W_{n,2}, \ldots, 
    W_{n,n}$ be another triangular array of random variables independent of 
    the $X_{n,i}$ (but not necessarily of each other).  Define 
    \(
        \bW_n = \frac{1}{n} \sum_{i=1}^n W_{n,i}.
    \) 
    If $\bW_n \toas \bW$ and 
    $\E \left[ \frac{1}{n} \sum_{i=1}^n W_{n,i}^4 \right]$ is uniformly 
    bounded in $n$, then
    \begin{equation}
        \frac{1}{n} \sum_{i=1}^n W_{n,i} \, X_{n,i} \toas \bW \mu.
    \end{equation}
\end{proposition}
\begin{proof}
    Define $S_n = \sum_{i=1}^n W_{n,i} \, X_{n,i}$ and let
    $\F^W_n = \sigma( W_{n,1}, W_{n,2}, \ldots, W_{n,n})$.  We have that
    \[
        \frac{1}{n} S_n - \bW_n \, \mu
        = 
       \frac{1}{n}
       \sum_{i=1}^n
           W_{n,i} \,
           (X_{n,i} - \mu) 
    \]
    Note that
    \begin{align*}
        \begin{split}
        \E \Bigg[
            \frac{1}{n^2}
            &\left(
                \sum_{i=1}^n
                    W_{n,i} \,
                    (X_{n,i} - \mu) 
            \right)^4
        \,\, \Bigg| \,\,
            \F^W_n
        \Bigg] \\
        &= \frac{1}{n^2} \left(
               \E \left[ X_{n,1} - \mu \right]^4
               \sum_{i=1}^n
                   W_{n,i}^4
               +
               3 \, \E \left[
                   (X_{n,1} - \mu)^2 (X_{n,2} - \mu)^2
               \right]
               \sum_{i \neq j}
                   W_{n,i}^2 W_{n,j}^2
           \right)
        \end{split} \\
        &\leq C_1 \left[
            \frac{1}{n}
            \sum_{i=1}^n W_{n,i}^2
        \right]^2
    \end{align*}
    for some constant $C_1$ bounding $\E \left[ X_{n,1} - \mu \right]^4$
    and $3 \, \E \left[ (X_{n,1} - \mu)^2 (X_{n,2} - \mu)^2 \right]$.
    Therefore, the full expectation is bounded by some other constant $C_2$.  
    We have just shown that
    \[
        \E \left[ 
            \frac{1}{n} S_n - \bW_n \, \mu
           \right]^4
        \leq \frac{C}{n^2}
    \]
    for some constant $C$.  Applying Chebyschev's inequality, we get
    \[
        P \left\{
            \left|
                \frac{1}{n} S_n - \bW_n \, \mu
            \right|
            > \varepsilon
        \right\}
        \leq \frac{C}{n^2 \varepsilon^4}.
    \]
    Invoking the first Borel-Cantelli Lemma, see the sum converges
    almost surely.
\end{proof}

Next, we derive a central limit theorem (CLT).  To prove it we will need a
CLT for dependent variables, which we take from McLeish \cite{mcleish1974dcl}:

\begin{lemma}\label{L:mcleigh}
    Let $X_{n,i}, \F_{n,i}, i = 1, \ldots, n$ be a martingale difference 
    array.  If the Lindeberg condition 
    $\sum_{i=1}^n \E \left[ X_{n,i}^2 ; | X_{n,i} | > \varepsilon \right] \to 0$
    is satisfied and $\sum_{i=1}^n X_{n,i}^2 \toP \sigma^2$ then
    $\sum_{i=1}^n X_{n,i} \tod \Normal (0, \, \sigma^2)$.
\end{lemma}

\noindent
With this Lemma, we can prove a CLT for weighted sums.

\begin{proposition}[Weighted CLT]\label{P:weighted-clt}
    Let $\vX_{n,i}, i = 1, \ldots, n$ be a triangular array of random vectors
    in $\reals^p$ with $\vX_{n,i}$ iid
    such that $\E \vX_{n,1} = \vmu^X$, $\cov[ \vX_{n,1}] = \mSigma^X$, and all 
    mixed fourth moments of the elements of $\vX_{n,1}$ are uniformly bounded in 
    $n$.  Let  $\vW_{n,i}$, $i = 1, \ldots, n$ be another triangular array 
    of random 
    vectors in $\reals^p$, independent of the $\vX_{n,i}$ but not necessarily 
    of each other.  Assume that 
    $\frac{1}{n} \sum_{i=1}^n \vW_{n,i} \vW_{n,i}^\trans \toP \mSigma^W$, and 
    that for
    all sets of indices $j_1, j_2, j_3, j_4$, with each index between $1$ and
    $p$, we have that
    \(
        \E \left[ 
            \frac{1}{n}
            \sum_{i=1}^n
                W_{n,ij_1} 
                W_{n,ij_2} 
                W_{n,ij_3} 
                W_{n,ij_4} 
        \right]
    \) is uniformly bounded in $n$.  Then
    \begin{equation}
        \sqrt{n}
        \left[
            \frac{1}{n}
            \sum_{i=1}^n
                \vW_{n,i} \bullet \vX_{n,i}
            -
            \frac{1}{n} 
            \sum_{i=1}^n 
                \vW_{n,i} \bullet \vmu^X
        \right]
        \tod
        \Normal\left( 0, \, \mSigma^W \bullet \mSigma^X \right),
    \end{equation}
    where $\bullet$ denotes Hadamard (elementwise) product.
\end{proposition}
\begin{proof}
    Let
    \[
        \vS_n
        =
        \sqrt{n} \left[ \frac{1}{n} \sum_{i=1}^n \vW_{n,i} \bullet \vX_{n,i}
            - \frac{1}{n} \sum_{i=1}^n \vW_{n,i} \bullet \vmu^X\right].
    \]
    We will use the Cram\'er-Wold device.  Let $\vtheta \in \reals^p$ be
    arbitrary.  Define
    \[
        Y_{n,i} 
        = 
        \frac{1}{\sqrt{n}}
        \sum_{j=1}^p
            \theta_j W_{n,ij} \, (X_{n,ij} - \mu^X_j),
    \]
    so that $\vtheta^\trans \vS_n = \sum_{i=1}^n Y_{n,i}$.  Also, with
    $\F_{n,i} = \sigma( Y_{n,1}, Y_{n,2}, \ldots, Y_{n,i-1})$, the collection
    $\{ Y_{n,i}, \F_{n,i} \}$ is a martingale difference array.  We will
    use Lemma~\ref{L:mcleigh} to prove the result.  First, we compute
    the variance as
    \begin{align*}
        \sum_{i=1}^n Y_{n,i}^2
        &=  \frac{1}{n}
            \sum_{i=1}^n
            \left[
                \sum_{j=1}^p
                    \theta_j W_{n,ij} \, (X_{n,ij} - \mu^X_j) 
            \right]^2 \\
        &=  \frac{1}{n}
            \sum_{i=1}^n
            \sum_{j=1}^n
            \sum_{k=1}^p
                \theta_j \theta_k
                W_{n,ij} W_{n,ik}
                (X_{n,ij} - \mu^X_j) (X_{n,ik} - \mu^X_k) \\
        &\toP \sum_{j=1}^p \sum_{k=1}^p
              \theta_j \theta_k
              \Sigma^W_{jk} \Sigma^X_{jk} \\
        &= \vtheta^\trans 
           \left(
               \mSigma^W
               \bullet
               \mSigma^X
           \right)
           \vtheta,
    \end{align*}
    where we have used  the fourth moment assumptions and
    Proposition~\ref{P:weighted-wlln} to get the convergence.  Lastly we check 
    the Lindeberg condition.  We have that
    \begin{align*}
        \sum_{i=1}^n
            &\E \left[ 
                Y_{n,i}^2 ; | Y_{n,i} | > \varepsilon
            \right] \\
        &\leq
            \frac{1}{\varepsilon^2}
            \sum_{i=1}^n
            \E \left[ 
                Y_{n,i}^4
            \right] \\
        &=
            \frac{1}{\varepsilon^2 n^2}
            \sum_{i=1}^n
            \sum_{j_1, \ldots, j_4}
                \Big\{
                 \theta_{j_1}
                 \theta_{j_2}
                 \theta_{j_3}
                 \theta_{j_4} 
                 \cdot \E \left[ 
                         W_{n,ij_1}
                         W_{n,ij_2}
                         W_{n,ij_3}
                         W_{n,ij_4}
                     \right] \\
                &\qquad\qquad\qquad\qquad\quad \cdot \E \Big[ 
                        (X_{n,ij_1} - \mu^X_{j_1})
                        (X_{n,ij_2} - \mu^X_{j_2})
                        (X_{n,ij_3} - \mu^X_{j_3})
                        (X_{n,ij_4} - \mu^X_{j_4})
                    \Big] \Big\} \\
        &\leq
            \frac{C_1}{\varepsilon^2 n}
            \sum_{j_1, \ldots, j_4}
                \E \left[
                    \frac{1}{n}
                    \sum_{i=1}^n
                        W_{n,ij_1}
                        W_{n,ij_2}
                        W_{n,ij_3}
                        W_{n,ij_4}
                \right] \\
        &\leq
            \frac{C_1 C_2 p^4}{\varepsilon^2 n} \\
        &\to 0
    \end{align*}
    where $C_1$ is a constant bounding $|\theta_{j}|^4$ and the
    centered fourth moments of $X_{n,ij}$, and $C_2$ bounds the
    fourth moments of the weights.
\end{proof}

For some classes of weights, we can get a stronger result.

\begin{corollary}\label{C:strong-weighted-clt}
    With the same assumptions as in Proposition~\ref{P:weighted-clt},
    if the mean of the weights converges sufficiently fast as
    \(
        \sqrt{n} \left[ \frac{1}{n} \sum_{i=1}^n \vW_{n,i} - \vmu^W \right] 
        \toP 
        0,
    \)
    then
    \begin{equation}
        \sqrt{n}
        \left[
            \frac{1}{n}
            \sum_{i=1}^n
                \vW_{n,i} \bullet \vX_{n,i}
            -
            \vmu^W \bullet \vmu^X
        \right]
        \tod
        \Normal\left( 0, \, \mSigma^W \bullet \mSigma^X \right).
    \end{equation}
\end{corollary}

perry-thesis.bbl
\begin{thebibliography}{10}

\bibitem{anderson1963atp}
T.W. Anderson.
\newblock {Asymptotic theory for principal component analysis}.
\newblock {\em Ann. Math. Stat.}, 34(1):122--148, 1963.

\bibitem{bai2002dnf}
J.~Bai and S.~Ng.
\newblock {Determining the number of factors in approximate factor models}.
\newblock {\em Econometrica}, 70(1):191--221, 2002.

\bibitem{bai2007ael}
Z.D. Bai, B.Q. Miao, and G.M. Pan.
\newblock {On asymptotics of eigenvectors of large sample covariance matrix}.
\newblock {\em Ann. Probab.}, 35(4):1532--1572, 2007.

\bibitem{bai2003crs}
Z.D. Bai, B.Q. Miao, and J.F. Yao.
\newblock {Convergence rates of the spectral distributions of large sample
  covariance matrices}.
\newblock {\em SIAM J. Matrix Anal. Appl.}, 25(1):105--127, 2003.

\bibitem{bai1998neo}
Z.D. Bai and J.W. Silverstein.
\newblock {No eigenvalues outside the support of the limiting spectral
  distribution of large dimensional sample covariance matrices}.
\newblock {\em Ann. Probab.}, 26(1):316--345, 1998.

\bibitem{bai2004clt}
Z.D. Bai and J.W. Silverstein.
\newblock {CLT for linear spectral statistics of large-dimensional sample
  covairance matrices}.
\newblock {\em Ann. Probab.}, 32(1A):553--605, 2004.

\bibitem{bai1988nle}
Z.D. Bai, J.W. Silverstein, and Y.Q. Yin.
\newblock {A note on the largest eigenvalue of a large dimensional sample
  covariance matrix}.
\newblock {\em J. Multivariate Anal.}, 26(2):166--168, 1988.

\bibitem{bai2008clt}
Z.D. Bai and J.F. Yao.
\newblock {Central limit theorems for eigenvalues in a spiked population
  model}.
\newblock {\em Ann. I. H. Poincar{\'e} -- PR}, 44(3):447--474, 2008.

\bibitem{bai2008lts}
Z.D. Bai and J.F. Yao.
\newblock {Limit theorems for sample eigenvalues in a generalized spiked
  population model}.
\newblock arXiv:0806.1141v1 [math.ST], 2008.

\bibitem{bai1993lse}
Z.D. Bai and Y.Q. Yin.
\newblock {Limit of the smallest eigenvalue of a large dimensional sample
  covariance matrix}.
\newblock {\em Ann. Probab.}, 21(3):1275--1294, 1993.

\bibitem{baik2005ptl}
J.~Baik, G.~Ben~Arous, and S.~P\'ech\'e.
\newblock {Phase transition of the largest eigenvalue for nonnull complex
  sample covariance matrices}.
\newblock {\em Ann. Probab.}, 33(5):1643--1697, 2005.

\bibitem{baik2006els}
J.~Baik and J.W. Silverstein.
\newblock {Eigenvalues of large sample covariance matrices of spiked population
  models}.
\newblock {\em J. Multivariate Anal.}, 97(6):1382--1408, 2006.

\bibitem{baker1998rme}
T.H. Baker, P.J. Forrester, and P.A. Pearce.
\newblock {Random matrix ensembles with an effective extensive external
  charge}.
\newblock {\em J. Phys. Math. Gen.}, 31(29):6087--6102, 1998.

\bibitem{bezdek1980fmp}
J.~C. Bezdek.
\newblock {\em Fuzzy Mathematics in Pattern Classification}.
\newblock PhD thesis, Cornell University, 1973.

\bibitem{breiman1992little}
L.~Breiman.
\newblock {The little bootstrap and other methods for dimensionality selection
  in regression: $X$-fixed prediction error}.
\newblock {\em J. Am. Stat. Assoc.}, 87(419):738--754, 1992.

\bibitem{breiman1984cart}
L.~Breiman, J.~Friedman, R.~Olshen, and C.~Stone.
\newblock {\em {Classification and Regression Trees}}.
\newblock Wadsworth, 1984.

\bibitem{breiman1992submodel}
L.~Breiman and P.~Spector.
\newblock {Submodel selection and evaluation in regression. The $X$-random
  case}.
\newblock {\em Int. Stat. Rev.}, 60(3):291--319, 1992.

\bibitem{cattell1966scree}
R.B. Cattell.
\newblock {The scree test for the number of factors}.
\newblock {\em Multivariate Behav. Res.}, 1(2):245--276, 1966.

\bibitem{chen2009ppc}
K.~Chen, D.~Paul, and J.L. Wang.
\newblock {Properties of principal component analysis for correlated data}.
\newblock Unpublished manuscript, 2009.

\bibitem{craven1978smoothing}
P.~Craven and G.~Wahba.
\newblock {Smoothing noisy data with spline functions: estimating the correct
  degree of smoothing by the method of generalized cross-validation}.
\newblock {\em Numer. Math.}, 31(4):377--403, 1979.

\bibitem{cutler1994aa}
A.~Cutler and L.~Breiman.
\newblock {Archetypal analysis}.
\newblock {\em Technometrics}, 36(4):338--347, 1994.

\bibitem{dempster1977maximum}
A.P. Dempster, N.M. Laird, and D.B. Rubin.
\newblock {Maximum likelihood from incomplete data via the EM algorithm}.
\newblock {\em J. Roy. Stat. Soc. B}, 39(1):1--38, 1977.

\bibitem{diaconis1987sag}
P.~Diaconis and M.~Shahshahani.
\newblock {The subgroup algorithm for generating uniform random variables}.
\newblock {\em Prob. Eng. Inform. Sc.}, 1(1):15--32, 1987.

\bibitem{diaconis1994erm}
P.~Diaconis and M.~Shahshahani.
\newblock {On the eigenvalues of random matrices}.
\newblock {\em J. Appl. Probab.}, 31A:49--62, 1994.

\bibitem{drineas2007fmc}
P.~Drineas, R.~Kannan, and M.W. Mahoney.
\newblock {Fast Monte Carlo algorithms for matrices III: Computing a compressed
  approximate matrix decomposition}.
\newblock {\em SIAM J. Comput.}, 36(1):184--206, 2006.

\bibitem{edelman1988ecn}
A.~Edelman.
\newblock {Eigenvalues and condition numbers of random matrices}.
\newblock {\em SIAM J. Matrix Anal. Appl.}, 9(4):543--560, 1988.

\bibitem{efron2008smi}
B.~Efron.
\newblock {Are a set of microarrays independent of each other?}
\newblock Unpublished manuscript, {2008}.

\bibitem{elkaroui2003lew}
N.~El~Karoui.
\newblock {On the largest eigenvalue of Wishart matrices with identity
  covariance when $n$, $p$ and $p/n$ tend to infinity}.
\newblock arXiv:math/0309355v1 [math.ST], 2003.

\bibitem{elkaroui2006mpt}
N.~El~Karoui.
\newblock {A rate of convergence result for the largest eigenvalue of complex
  white Wishart matrices}.
\newblock {\em Ann. Probab.}, 34(6):2077--2117, 2006.

\bibitem{elkaroui2007twl}
N.~El~Karoui.
\newblock {Tracy-Widom limit for the largest eigenvalue of a large class of
  complex sample covariance matrices}.
\newblock {\em Ann. Probab.}, 35(2):663--714, 2007.

\bibitem{faber1997critical}
K.~Faber and B.R. Kowalski.
\newblock {Critical evaluation of two $F$-tests for selecting the number of
  factors in abstract factor analysis}.
\newblock {\em Anal. Chim. Acta}, 337(1):57--71, 1997.

\bibitem{fazel2002mrm}
M.~Fazel.
\newblock {\em {Matrix Rank Minimization with Applications}}.
\newblock PhD thesis, Stanford University, 2002.

\bibitem{fisher1923scv}
R.A. Fisher and W.A. Mackenzie.
\newblock {Studies in crop variation. II. The manurial response of different
  potato varieties}.
\newblock {\em J. Agric. Sci.}, 13:311--320, 1923.

\bibitem{geman1980ltn}
S.~Geman.
\newblock {A limit theorem for the norm of random matrices}.
\newblock {\em Ann. Probab.}, 8(2):252--261, 1980.

\bibitem{golub1979generalized}
G.H. Golub, M.~Heath, and G.~Wahba.
\newblock {Generalized cross-validation as a method for choosing a good ridge
  parameter}.
\newblock {\em Technometrics}, 21(2):215--223, 1979.

\bibitem{golub1996mc}
G.H. Golub and C.F. Van~Loan.
\newblock {\em {Matrix Computations}}.
\newblock Johns Hopkins University Press, 1996.

\bibitem{grenander1977san}
U.~Grenander and J.W. Silverstein.
\newblock {Spectral analysis of networks with random topologies}.
\newblock {\em SIAM J. Appl. Math.}, 32(2):499--519, 1977.

\bibitem{guionnetlds}
A.~Guionnet.
\newblock {Large deviations and stochastic calculus for large random matrices}.
\newblock {\em Probab. Surv.}, 1:72--172, 2004.

\bibitem{guionnet2000csm}
A.~Guionnet and O.~Zeitouni.
\newblock {Concentration of the spectral measure for large matrices}.
\newblock {\em Electro. Comm. Probab.}, 5:119--136, 2000.

\bibitem{hastings1980bvp}
S.~P. Hastings and J.~B. McLeod.
\newblock A boundary value problem associated with the second {P}ainlev\'e
  transcendent and the {K}orteweg-{d}e {V}ries equation.
\newblock {\em Archive Rat. Mech. Anal.}, 73(1):31--51, 1980.

\bibitem{hiai1998edw}
F.~Hiai and D.~Petz.
\newblock {Eigenvalue density of the Wishart matrix and large deviations}.
\newblock {\em Inf. Dim. Anal. Quantum Probab. Rel. Top}, 1(4):633--646, 1998.

\bibitem{jackson1993stopping}
D.A. Jackson.
\newblock {Stopping rules in principal components analysis: a comparison of
  heuristical and statistical approaches}.
\newblock {\em Ecology}, 74(8):2204--2214, 1993.

\bibitem{james1961ewq}
W.~James and C.~Stein.
\newblock {Estimation with quadratic loss}.
\newblock In {\em Proc. Fourth Berkeley Symp. Math. Statist. Prob.}, pages
  361--380. Univ. of California Press, 1960.

\bibitem{johansson2000sfa}
K.~Johansson.
\newblock {Shape fluctuations and random matrices}.
\newblock {\em Comm. Math. Phys.}, 209(2):437--476, 2000.

\bibitem{johnstone2001dle}
I.M. Johnstone.
\newblock {On the distribution of the largest eigenvalue in principal
  components analysis}.
\newblock {\em Ann. Statist.}, 29(2):295--327, 2001.

\bibitem{jolliffe2002pca}
I.T. Jolliffe.
\newblock {\em {Principal Component Analysis}}.
\newblock Springer New York, 2002.

\bibitem{jonsson1982slt}
D.~Jonsson.
\newblock {Some limit theorems for the eigenvalues of a sample covariance
  matrix}.
\newblock {\em J. Multivariate Anal.}, 12(1):1--38, 1982.

\bibitem{jonsson1983ole}
D.~Jonsson.
\newblock {On the largest eigenvalue of a sample covariance matrix}.
\newblock In P.R. Krishnaiah, editor, {\em Multivariate Analysis VI}.
  North-Holland, 1983.

\bibitem{kolda1998smd}
T.G. Kolda and D.P. O'Leary.
\newblock {A semidiscrete matrix decomposition for latent semantic indexing
  information retrieval}.
\newblock {\em ACM Trans. Inform. Syst.}, 16(4):322--346, 1998.

\bibitem{kritchman2008determining}
S.~Kritchman and B.~Nadler.
\newblock {Determining the number of components in a factor model from limited
  noisy data}.
\newblock {\em Chemometr. Intell. Lab. Syst.}, 94(1):19--32, 2008.

\bibitem{lazzeroni2002pmg}
L.~Lazzeroni and A.~Owen.
\newblock {Plaid models for gene expression data}.
\newblock {\em Stat. Sinica}, 12(1):61--86, 2002.

\bibitem{lee1999lpo}
D.D. Lee and H.S. Seung.
\newblock {Learning the parts of objects by non-negative matrix factorization}.
\newblock {\em Nature}, 401(6755):788--791, 1999.

\bibitem{ma2008atw}
Z.~Ma.
\newblock {Accuracy of the Tracy-Widom limit for the largest eigenvalue in
  white Wishart matrices}.
\newblock arXiv:0810.1329v1 [math.ST], 2008.

\bibitem{malinowski1989statisticalf}
E.R. Malinowski.
\newblock {Statistical $F$-tests for abstract factor analysis and target
  testing}.
\newblock {\em J. Chemometr.}, 3(1):49--69, 1989.

\bibitem{manning1999fsn}
C.D. Manning, H.~Sch{\"u}tze, and MIT Press.
\newblock {\em {Foundations of Statistical Natural Language Processing}}.
\newblock MIT Press, 1999.

\bibitem{marcenko1967des}
V.A. Mar{\v{c}}enko and L.A. Pastur.
\newblock {Distribution of eigenvalues for some sets of random matrices}.
\newblock {\em Sb. Math.}, 1(4):457--483, 1967.

\bibitem{markowitz1952ps}
H.~Markowitz.
\newblock {Portfolio selection}.
\newblock {\em J. Finance}, 7(1):77--91, 1952.

\bibitem{mcleish1974dcl}
D.L. McLeish.
\newblock {Dependent central limit theorems and invariance principles}.
\newblock {\em Ann. Probab.}, 2(4):620--628, 1974.

\bibitem{meckes2006ivs}
E.~Meckes.
\newblock {\em {An Infinitesimal Version of Stein's Method of Exchangeable
  Pairs}}.
\newblock PhD thesis, Stanford University, 2006.

\bibitem{mezzadri2007grm}
F.~Mezzadri.
\newblock {How to generate random matrices from the classical compact groups}.
\newblock {\em Notices Amer. Math. Soc.}, 54(5):592, 2007.

\bibitem{muirhead1982ams}
R.J. Muirhead.
\newblock {\em {Aspects of Multivariate Statistical Theory}}.
\newblock John Wiley \& Sons, 1982.

\bibitem{nadler2008fsa}
B.~Nadler.
\newblock {Finite sample approximation results for principal component
  analysis: A matrix perturbation approach}.
\newblock {\em Ann. Statist.}, 36(6):2791--2817, 2008.

\bibitem{okamoto1973deq}
M.~Okamoto.
\newblock {Distinctness of the eigenvalues of a quadratic form in a
  multivariate sample}.
\newblock {\em Ann. Statist.}, 1(4):763--765, 1973.

\bibitem{onatski2009}
A.~Onatski.
\newblock {Asymptotic distribution of the principal components estimator of
  large factor models when factors are relatively weak}.
\newblock Unpublished manuscript, {2009}.

\bibitem{owen2009bi}
A.B. Owen and P.O. Perry.
\newblock {Bi-cross-validation of the SVD and the non-negative matrix
  factorization}.
\newblock {\em Ann. Appl. Statist.}, 3(2):564--594, 2009.

\bibitem{pastur2008clt}
L.~Pastur and A.~Lytova.
\newblock {Central limit theorem for linear eigenvalue statistics of random
  matrices with independent entries}.
\newblock arXiv:0809.4698v1 [math.PR], 2008.

\bibitem{paul2006dse}
D.~Paul.
\newblock {Distribution of the smallest eigenvalue of a Wishart($N$,$n$) when
  $N/n \to 0$}.
\newblock Unpublished manuscript, 2006.

\bibitem{paul2007ase}
D.~Paul.
\newblock {Asymptotics of sample eigenstructure for a large dimensional spiked
  covariance model}.
\newblock {\em Stat. Sinica}, 17(4):1617--1642, 2007.

\bibitem{peche2008url}
S.~P\'ech\'e.
\newblock {Universality results for largest eigenvalues of some sample
  covariance matrix ensembles}.
\newblock arXiv:0705.1701v2 [math.PR], 2008.

\bibitem{perry2009mre}
P.O. Perry and P.J. Wolfe.
\newblock {Minimax rank estimation for subspace tracking}.
\newblock arXiv:0906.3090v1 [stat.ME], 2009.

\bibitem{rao2008sample}
N.R. Rao and A.~Edelman.
\newblock {Sample eigenvalue based detection of high-dimensional signals in
  white noise using relatively few samples}.
\newblock {\em IEEE Trans. Signal Process.}, 56(7 Part 1):2625--2638, 2008.

\bibitem{roy1953heuristic}
S.N. Roy.
\newblock {On a heuristic method of test construction and its use in
  multivariate analysis}.
\newblock {\em Ann. Math. Stat.}, 24(2):220--238, 1953.

\bibitem{schena1995qmg}
M.~Schena, D.~Shalon, R.W. Davis, and P.O. Brown.
\newblock {Quantitative monitoring of gene expression patterns with a
  complementary DNA microarray}.
\newblock {\em Science}, 270(5235):467--470, 1995.

\bibitem{silverstein1979reg}
J.W. Silverstein.
\newblock {On the randomness of eigenvectors generated from networks with
  random topologies}.
\newblock {\em SIAM J. Appl. Math.}, 37(2):235--245, 1979.

\bibitem{silverstein1981dbe}
J.W. Silverstein.
\newblock {Describing the behavior of eigenvectors of random matrices using
  sequences of measures on orthogonal groups}.
\newblock {\em SIAM J. Math. Anal.}, 12(2):274--281, 1981.

\bibitem{silverstein1984ole}
J.W. Silverstein.
\newblock {On the largest eigenvalue of a large dimensional sample covariance
  matrix}.
\newblock Unpublished manuscript, 1984.

\bibitem{silverstein1984slt}
J.W. Silverstein.
\newblock {Some limit theorems on the eigenvectors of large dimensional sample
  covariance matrices}.
\newblock {\em J. Multivariate Anal.}, 15(3):295--324, 1984.

\bibitem{silverstein1985sel}
J.W. Silverstein.
\newblock {The smallest eigenvalue of a large dimensional Wishart matrix}.
\newblock {\em Ann. Probab.}, 13(4):1364--1368, 1985.

\bibitem{silverstein1989eld}
J.W. Silverstein.
\newblock {On the eigenvectors of large dimensional sample covariance
  matrices}.
\newblock {\em J. Multivariate Anal.}, 30(1):1--16, 1989.

\bibitem{silverstein1990wcr}
J.W. Silverstein.
\newblock {Weak convergence of random functions defined by the eigenvectors of
  sample covariance matrices}.
\newblock {\em Ann. Probab.}, 18(3):1174--1194, 1990.

\bibitem{silverstein1995sce}
J.W. Silverstein.
\newblock {Strong convergence of the emperical distribution of eigenvalues of
  large dimensional random matrices}.
\newblock {\em J. Multivariate Anal.}, 55(2):331--229, 1995.

\bibitem{silverstein1995ede}
J.W. Silverstein and Z.D. Bai.
\newblock {On the empirical distribution of eigenvalues of a class of large
  dimensional random matrices}.
\newblock {\em J. Multivariate Anal.}, 54(2):175--192, 1995.

\bibitem{soshnikov2002nud}
A.~Soshnikov.
\newblock {A note on universality of the distribution of the largest
  eigenvalues in certain sample covariance matrices}.
\newblock {\em J. Stat. Phys.}, 108(5):1033--1056, 2002.

\bibitem{stewart1990pts}
G.W. Stewart.
\newblock {Perturbation theory for the singular value decomposition}.
\newblock Technical Report CS-TR-2539, University of Maryland, September 1990.

\bibitem{stewart1990mpt}
G.W. Stewart and J.~Sun.
\newblock {\em {Matrix Perturbation Theory}}.
\newblock Academic Press, 1990.

\bibitem{stone1974cv}
M.~Stone.
\newblock {Cross-validatory choice and assessment of statistical predictions}.
\newblock {\em J. Roy. Stat. Soc. B}, 36(2):111--147, 1974.

\bibitem{tao2009rmd}
T.~Tao and V.~Vu.
\newblock {Random matrices: The distribution of the smallest singular values}.
\newblock arXiv:0903.0614v1 [math.PR], 2009.

\bibitem{tibshirani2009}
R.~Tibshirani and R.~Tibshirani.
\newblock {A bias correction for the minimum error rate in cross-validation}.
\newblock {\em Ann. Appl. Statist.}, 3(2):822--829, 2009.

\bibitem{tracy1994lsd}
C.A. Tracy and H.~Widom.
\newblock {Level-spacing distributions and the Airy kernel}.
\newblock {\em Comm. Math. Phys.}, 159(1):151--174, 1994.

\bibitem{tracy1996oas}
C.A. Tracy and H.~Widom.
\newblock {On orthogonal and symplectic matrix ensembles}.
\newblock {\em Comm. Math. Phys.}, 177(3):727--754, 1996.

\bibitem{troyanskaya2001missing}
O.~Troyanskaya, M.~Cantor, G.~Sherlock, P.~Brown, T.~Hastie, R.~Tibshirani,
  D.~Botstein, and R.B. Altman.
\newblock {Missing value estimation methods for DNA microarrays}.
\newblock {\em Bioinformatics}, 17(6):520--525, 2001.

\bibitem{wachter1978slr}
K.W. Wachter.
\newblock {The strong limits of random matrix spectra for sample matrices of
  independent elements}.
\newblock {\em Ann. Probab.}, 6(1):1--18, 1978.

\bibitem{wax1985detection}
M.~Wax and T.~Kailath.
\newblock {Detection of signals by information theoretic criteria}.
\newblock {\em IEEE Trans. Acoust., Speech, Signal Process.}, 33(2):387--392,
  1985.

\bibitem{wold1969nonlinear}
H.~Wold and E.~Lyttkens.
\newblock {Nonlinear iterative partial least squares (NIPALS) estimation
  procedures}.
\newblock In {\em Bull. Intern. Statist. Inst: Proc. 37th Session, London},
  pages 1--15, 1969.

\bibitem{wold1978cross}
S.~Wold.
\newblock {Cross-validatory estimation of the number of components in factor
  and principal components models}.
\newblock {\em Technometrics}, 20(4):397--405, 1978.

\bibitem{yin1986lsd}
Y.Q. Yin.
\newblock {Limiting spectral distribution for a class of random matrices}.
\newblock {\em J. Multivariate Anal.}, 20(1):50--68, 1986.

\bibitem{yin1988lle}
Y.Q. Yin, Z.D. Bai, and P.R. Krishnaiah.
\newblock {On the limit of the largest eigenvalue of the large dimensional
  sample covariance matrix}.
\newblock {\em Probab. Theor. Relat. Field.}, 78(4):509--521, 1988.

\bibitem{yin1983lte}
Y.Q. Yin and P.R. Krishnaiah.
\newblock {A limit theorem for the eigenvalues of product of two random
  matrices}.
\newblock {\em J. Multivariate Anal.}, 13(4):489--507, 1983.

\end{thebibliography}
