Testing in high-dimensional spiked models
Iain M. Johnstone and Alexei Onatski.
We consider five different classes of multivariate statistical problems identified by James (1964). Each of these problems is related to the eigenvalues of $E^{-1H$ where $H$ and $E$ are proportional to high-dimensional Wishart matrices. Under the null hypothesis, both Wisharts are central with identity covariance. Under the alternative, the non-centrality or the covariance parameter of $H$ has a single eigenvalue, a spike, that stands alone. When the spike is larger than a case-specific phase transition threshold, one of the eigenvalues of $E^{-1H$ separates from the bulk. This makes the alternative easily detectable, so that reasonable statistical tests have asymptotic power one. In contrast, when the spike is sub-critical, that is lies below the threshold, none of the eigenvalues separates from the bulk, which makes the testing problem more interesting from the statistical perspective. In such cases, we show that the log likelihood ratio processes parameterized by the value of the sub-critical spike converge to Gaussian processes with logarithmic correlation. We use this result to derive the asymptotic power envelopes for tests for the presence of a spike in the data representing each of the five cases in James' classification.
Subcritical20.tex
%2multibyte Version: 5.50.0.2960 CodePage: 65001
\documentclass[12pt]{article}%
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage[onehalfspacing]{setspace}
\usepackage{geometry}%
\setcounter{MaxMatrixCols}{30}
%TCIDATA{OutputFilter=latex2.dll}
%TCIDATA{Version=5.50.0.2960}
%TCIDATA{Codepage=65001}
%TCIDATA{CSTFile=40 LaTeX article.cst}
%TCIDATA{Created=Thursday, July 03, 2014 12:44:26}
%TCIDATA{LastRevised=Monday, September 21, 2015 21:43:17}
%TCIDATA{<META NAME="GraphicsSave" CONTENT="32">}
%TCIDATA{<META NAME="SaveForMode" CONTENT="1">}
%TCIDATA{BibliographyScheme=Manual}
%TCIDATA{<META NAME="DocumentShell" CONTENT="Standard LaTeX\Blank - Standard LaTeX Article">}
%TCIDATA{Language=American English}
%TCIDATA{ComputeDefs=
%$f^{(1)}\left(  z\right)  =-\frac{z}{2c_{p}}\frac{\theta}{1+\theta}$
%}
%BeginMSIPreambleData
\providecommand{\U}[1]{\protect\rule{.1in}{.1in}}
%EndMSIPreambleData
\providecommand{\U}[1]{\protect\rule{.1in}{.1in}}
\newtheorem{theorem}{Theorem}
\newtheorem{acknowledgement}[theorem]{Acknowledgement}
\newtheorem{algorithm}[theorem]{Algorithm}
\newtheorem{axiom}[theorem]{Axiom}
\newtheorem{case}[theorem]{Case}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{conclusion}[theorem]{Conclusion}
\newtheorem{condition}[theorem]{Condition}
\newtheorem{conjecture}[theorem]{Conjecture}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{criterion}[theorem]{Criterion}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{example}[theorem]{Example}
\newtheorem{exercise}[theorem]{Exercise}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{notation}[theorem]{Notation}
\newtheorem{problem}[theorem]{Problem}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{remark}[theorem]{Remark}
\newtheorem{solution}[theorem]{Solution}
\newtheorem{summary}[theorem]{Summary}
\newenvironment{proof}[1][Proof]{\noindent\textbf{#1.} }{\ \rule{0.5em}{0.5em}}
\geometry{left=1.5in,right=1.2in,top=1in,bottom=1.1in}
\begin{document}

\title{Testing in high-dimensional spiked models}
\author{Iain M. Johnstone and Alexei Onatski.}
\maketitle

\begin{abstract}
We consider five different classes of multivariate statistical problems
identified by James (1964). Each of these problems is related to the
eigenvalues of $E^{-1}H$ where $H$ and $E$ are proportional to
high-dimensional Wishart matrices. Under the null hypothesis, both Wisharts
are central with identity covariance. Under the alternative, the
non-centrality or the covariance parameter of $H$ has a single eigenvalue, a
spike, that stands alone. When the spike is larger than a case-specific phase
transition threshold, one of the eigenvalues of $E^{-1}H$ separates from the
bulk. This makes the alternative easily detectable, so that reasonable
statistical tests have asymptotic power one. In contrast, when the spike is
sub-critical, that is lies below the threshold, none of the eigenvalues
separates from the bulk, which makes the testing problem more interesting from
the statistical perspective. In such cases, we show that the log likelihood
ratio processes parameterized by the value of the sub-critical spike converge
to Gaussian processes with logarithmic correlation. We use this result to
derive the asymptotic power envelopes for tests for the presence of a spike in
the data representing each of the five cases in James' classification.

\end{abstract}

\section{Introduction}

High-dimensional multivariate models and methods, such as regression,
principal components, and canonical correlation analysis, have become subject
of much recent research. In contrast to the classical framework where the
dimensionality is fixed, the current focus is on situations where the
dimensionality diverges to infinity together with the sample size. In this
context, \textit{spiked models} that deviate from a reference model along a
small fixed number of unknown directions have proven to be a fruitful research
tool. A basic statistical question that arises in the analysis of spiked
models is how to test for the presence of spikes in the data.

James (1964) arranges multivariate statistical problems in five different
groups with broadly similar features. His classification corresponds to the
five types of the hypergeometric functions $_{p}F_{q}$ that often occur in
multivariate distributions. In this paper, we describe spiked models that
represent each of James' classes, and derive the asymptotic behavior of the
corresponding likelihood ratios, that is the ratios of the joint densities of
the relevant data under the alternative hypothesis, which assumes the presence
of the spikes, to that under the null of no spikes. In each of the cases, the
relevant data consist of the maximal invariant statistic represented by
eigenvalues of a large random matrix. We consider the asymptotic regime where
the dimensionality of the data and the number of observations go to infinity proportionally.

We find that the measures corresponding to the joint distributions of the
eigenvalues under the alternative hypothesis and under the null are mutually
contiguous when the values of the spikes are below a phase transition
threshold. The value of the threshold depends on the problem's type.
Furthermore, we find that the log likelihood ratio processes parametrized by
the values of the spikes are asymptotically Gaussian, with logarithmic mean
and autocovariance functions. These findings allow us to compute the
asymptotic power envelopes for the tests for the presence of spikes in five
multivariate models representing each of James' classes.

Our analysis is based on the classical results that assume Gaussianity. All
the likelihood ratios that we study correspond to the joint densities of the
solutions to the basic equation of classical multivariate statistics,%
\begin{equation}
\det\left(  H-\lambda E\right)  =0,\label{basic}%
\end{equation}
where $H$ and $E$ are proportional to Wishart matrices.

The five different cases that we study are: 1) $E$ is a known deterministic
matrix, and $H$ is a central Wishart matrix with covariance equal to a
low-rank perturbation of $E$; 2) both $E$ and $H$ are central Wisharts with
unknown covariance matrices that differ by a matrix of low rank; 3) $E$ is a
known deterministic matrix, and $H$ is a non-central Wishart matrix with
covariance equal to $E$ and with a low-rank non-centrality; 4) $E$ is a
central Wishart matrix, while $H$ is a non-central one with the same unknown
covariance matrix and with a low-rank non-centrality; 5) $E$ is a central
Wishart, while $H$ is a non-central Wishart conditionally on a random low-rank
non-centrality parameter. These five cases can be linked via sufficiency and
invariance arguments to a principal components problem, a signal detection
problem, hypotheses testing in multivariate regression with known and with
unknown error covariance, and a canonical correlation problem, respectively.
We briefly discuss the links in the next section of the paper.

The main steps of our asymptotic analysis are the same for all the five cases.
The likelihood ratios have explicit forms that involve hypergeometric
functions of two high-dimensional matrix arguments. However, the low-rank
nature of the alternatives that we consider ensures that one of the arguments
have low rank. For tractability, we focus on the special case of rank-one
alternatives. In such case, using the recent result of Dharmawansa and
Johnstone (2014), we represent the hypergeometric function of two
high-dimensional matrix arguments in the form of a contour integral that
involves a scalar hypergeometric function of the same type. Then we deform the
contour of integration so that the integral becomes amenable to Laplace
approximation analysis (see Olver (1997), chapter 4).

Using the Laplace approximation technique, we show that the log likelihood
ratios are asymptotically equivalent to random quadratic functions of the
spike parameters. The randomness in the quadratic function enters via a linear
spectral statistic of a large random matrix of either sample covariance or
$F$-ratio type. Using CLT for the linear spectral statistics, established by
Bai and Silverstein (2004) for the sample-covariance-type random matrices and
by Zheng (2012) for the $F$-ratio-type random matrices, we derive the
asymptotic Gaussianity and obtain the mean and the autocovariance functions of
the log likelihood ratio processes.

The derived asymptotics of the log likelihood processes shows that the
corresponding statistical experiments do not converge to Gaussian shift
experiments. In other words, the experiments that consist of observing the
solutions to equation (\ref{basic}) parameterized by the values of the spikes
under the alternative hypothesis are not of the Locally Asymptotically Normal
(LAN) type. This implies that there are no ready-to-use\ optimality results
associated with LAN experiments that can be applied in our setting. However at
the fundamental level, the derived asymptotics of the log likelihood ratio
processes is all that is needed for the asymptotic analysis of the risk of the
corresponding statistical decisions.

In this paper, we use the derived asymptotics together with the Neyman-Pearson
lemma and Le Cam's third lemma (see van der Vaart (1998)), to find simple
analytic expressions for the asymptotic power envelopes for the statistical
tests of the null hypothesis of no spikes in the data. The form of the
envelope is different depending on whether both $H$ and $E$ in the
corresponding equation (\ref{basic}) are Wisharts or only $H$ is Wishart
whereas $E$ is deterministic.

For most of the cases, as the value of the spike under the alternative
increases, the envelope, at first, rises very slowly. Then, as the spike
approaches the phase transition, the rise quickly accelerates and the envelope
`hits' unity at the threshold. However, in cases of two Wisharts and when the
dimensionality is not much smaller than the degrees of freedom of $E,$ the
envelope rises much faster. In such cases, the information in all the
eigenvalues of $E^{-1}H$ might be useful for detecting population spikes which
lie far below the phase transition threshold.

A type of the analysis performed in this paper has been previously implemented
in the study of the principal components case by Onatski et al (2013). Our
work extends theirs to the remaining four cases in James' classification of
multivariate statistical problems. One of the hardest challenges in such an
extension is the rigorous implementation of the Laplace approximation step.
With this goal in mind, we have developed asymptotic approximations to the
hypergeometric functions $_{1}F_{1}$ and $_{2}F_{1}$ which are uniform in
certain domains of the complex plane.

A trivial observation that the solutions to equation (\ref{basic}) can be
interpreted as the eigenvalues of random matrix $E^{-1}H$ relates our work to
the vast literature on the spectrum of large random matrices. We refer the
reader to Bai and Silverstein (2006) for a recent book-long treatment of the
subject. Three extensively studied classical ensembles of random matrices are
the Gaussian, Laguerre and Jacobi ensembles (see Mehta (2004)). However, only
the Laguerre and Jacobi ensembles are relevant for the five scenarios for
(\ref{basic}) that correspond to James' five-fold classification of
multivariate statistical problems. This prompts us to search for a
\textquotedblleft missing\textquotedblright\ class in James' system that could
be linked to the Gaussian ensemble.

Such a class is easy to obtain by taking the limit of $\sqrt{n_{1}}\left(
H-I_{p}\right)  $ as $n_{1}\rightarrow\infty,$ where $n_{1}$ and $p$ are $H$'s
degrees of freedom and dimensionality, respectively. The corresponding
statistical problem can be called \textquotedblleft symmetric matrix
denoising\textquotedblright. Under the null hypothesis, the observations are
given by a $p\times p$ matrix $Z/\sqrt{p}$ with $Z$ from the Gaussian
Orthogonal Ensemble. Under the alternative, the observations are given by
$Z/\sqrt{p}+\Phi,$ where $\Phi$ is a deterministic symmetric matrix of low
rank. We call this situation \textquotedblleft case zero\textquotedblright,
and add it to James' classification. We derive the asymptotics of the
corresponding log likelihood ratio and obtain the related asymptotic power envelope.

Many existing results in the random matrix literature do not require that the
data are Gaussian. This suggests that some results about tests for the
presence of the spikes in the data may remain valid without the Gaussianity.
One may for example consider $H$ and $E$ in (\ref{basic}) that, although have
the form of sample covariance matrices, do not come from the underlying
Gaussian distribution, and study the properties of the corresponding tests. We
leave this line of research to the future.

Since the explicit form of the joint distribution of the solutions to
(\ref{basic}) is only known in the Gaussian case, it seems unlikely that one
would be able to completely summarize the asymptotic behavior of the
corresponding non-Gaussian statistical experiments. We hope that the results
of this paper, that provide such a summary under the Gaussianity, can serve as
a useful benchmark for the future studies that would relax our assumptions.

The rest of the paper is organized as follows. In the next section, we relate
the five different cases of equation (\ref{basic}) to the classical
multivariate statistical problems representing different cells of James'
(1964) five-fold classification system. In Section 3, we obtain explicit
expressions for the likelihood ratios. Section 4 represents the likelihood
ratios in the form of contour integrals. Section 5 performs the Laplace
approximation analysis. Section 6 derives the asymptotic power envelopes.
Section 7 concludes. Technical proofs are given in the Appendix.

\section{Links to classical statistical problems}

Case 1 corresponds to the problem of using $n_{1}$ i.i.d. $N_{p}\left(
0,\Omega\right)  $ ($p$-dimensional Gaussian) observations to test the null
hypothesis that the population covariance $\Omega$ equals a given matrix
$\Sigma$. The alternative of interest is
\[
\Omega=\Sigma+\mathbb{\psi}\theta\mathbb{\psi}^{\prime}%
\]
with unknown $\theta>0$ and $\mathbb{\psi}$, where $\mathbb{\psi}$ is
normalized so that $\left\Vert \Sigma^{-1/2}\mathbb{\psi}\right\Vert =1$.

Without loss of generality, we may assume that $\Sigma=I_{p}$. Then under the
null, the data are isotropic noise, whereas under the alternative, the first
principal component explains a larger portion of the variation than the other
principal components. We therefore label Case 1 as the `principal components
analysis' (PCA) case.

The null and the alternative hypotheses can be formulated in terms of the
spectral `spike' parameter $\theta$ as
\begin{equation}
H_{0}:\theta_{0}=0\text{ and }H_{1}:\theta_{0}=\theta>0,
\label{Basic hypotheses}%
\end{equation}
where $\theta_{0}$ is the true value of the `spike'. This testing problem
remains invariant under the multiplication of the $p\times n_{1}$ data matrix
from the left and from the right by orthogonal matrices, and under the
corresponding transformation in the parameter space. A maximal invariant
statistic consists of the solutions $\lambda_{1}\geq...\geq\lambda_{p}$ of
equation (\ref{basic}) with $H$ equal to the sample covariance matrix and
$E=\Sigma$. We restrict attention to the invariant tests. Therefore, the
relevant data are summarized by $\lambda_{1},...,\lambda_{p}$.

Case 2 is represented by the problem of testing the equality of covariance
matrices, $\Omega$ and $\Sigma$, corresponding to two independent
$p$-dimensional zero-mean Gaussian samples of sizes $n_{1}$ and $n_{2}$.
Throughout the paper, we shall assume that
\[
p\leq\min\left\{  n_{1},n_{2}\right\}  .
\]
The assumption $p\leq n_{2}$ is made to ensure the almost sure invertibility
of matrix $E$ in (\ref{basic}), whereas the assumption $p\leq n_{1}$ is made
to reduce the number of various situations which need to be considered. Such a
reduction makes our exposition more concise.

Returning to Case 2, the alternative hypothesis is the same as in the PCA
case. Similar invariance considerations lead to tests based on the eigenvalues
of the $F$-ratio of the sample covariance matrices. Matrix $H$ from
(\ref{basic}) equals the sample covariance corresponding to the observations
that might contain a `signal' responsible for the covariance spike, whereas
matrix $E$ equals the other sample covariance matrix. We label Case 2 as the
`signal detection' (SigD) case. In this case, we find it more convenient to
work with the $p$ solutions to the equation%
\begin{equation}
\det\left(  H-\lambda\left(  E+\frac{n_{1}}{n_{2}}H\right)  \right)  =0,
\label{basic beta form}%
\end{equation}
which we also denote $\lambda_{1}\geq...\geq\lambda_{p}$ to make the notations
as uniform across the different cases as possible. Note that as the number of
observations in the second sample, $n_{2}$, diverges to infinity while $n_{1}$
and $p$ are held constant, equation (\ref{basic beta form}) reduces to
equation (\ref{basic}), $E$ converges to $\Sigma$, and SigD reduces to PCA.

Cases 3 and 4 occur in multivariate regression
\[
Y=X\beta+\varepsilon
\]
when the goal is to test linear restrictions on the matrix of coefficients
$\beta$. Case 3 corresponds to the situation where the covariance matrix
$\Sigma$ of the i.i.d. Gaussian rows of the error matrix $\varepsilon$ is
known. We label this case as `regression with known variance' (REG$_{0}$).
Case 4 corresponds to the unknown $\Sigma,$ and we label it as `regression
with unknown variance' (REG).

As explained in Muirhead (1982), pp. 433-434, the problem of testing linear
restrictions on $\beta$ can be cast in the canonical form, where the matrix of
transformed response variables is split into three parts, $Y_{1}^{\ast},$
$Y_{2}^{\ast},$ and $Y_{3}^{\ast}$. Matrix $Y_{1}^{\ast}$ is $n_{1}\times p,$
where $p$ is the number of response variables and $n_{1}$ is the number of
restrictions. Under the null hypothesis, $\mathbb{E}Y_{1}^{\ast}=0,$ whereas
under the alternative,
\begin{equation}
\mathbb{E}Y_{1}^{\ast}=\sqrt{n_{1}\theta}\varphi\psi^{\prime},
\label{REG0mean}%
\end{equation}
where $\theta>0,$ $\left\Vert \Sigma^{-1/2}\psi\right\Vert =1,$ and
$\left\Vert \varphi\right\Vert =1.$ Matrices $Y_{2}^{\ast}$ and $Y_{3}^{\ast}$
are $\left(  q-n_{1}\right)  \times p$ and $\left(  T-q\right)  \times p,$
respectively, where $q$ is the number of regressors and $T$ is the number of
observations. These matrices have, respectively, unrestricted and zero means
under both the null and the alternative.

For REG$_{0}$, sufficiency and invariance arguments lead to tests based on the
solutions $\lambda_{1},...,\lambda_{p}$ of (\ref{basic}) with
\[
H=Y_{1}^{\ast\prime}Y_{1}^{\ast}/n_{1}\text{ and }E=\Sigma.
\]
These solutions represent a multivariate analog of the difference between the
sum of squared residuals in the restricted and unrestricted regressions. For
REG, similar arguments lead to tests based on the $p$ solutions $\lambda
_{1},...,\lambda_{p}$ of (\ref{basic beta form}) with
\[
H=Y_{1}^{\ast\prime}Y_{1}^{\ast}/n_{1}\text{ and }E=Y_{3}^{\ast\prime}%
Y_{3}^{\ast}/n_{2},
\]
where $n_{2}=T-q$. These solutions represent a multivariate analog of the
ratio of the difference between the sum of squared residuals in the restricted
and unrestricted regressions to the sum of squared residuals in the restricted
regression. Note that, as $n_{2}\rightarrow\infty$ while $n_{1}$ and $p$ are
held constant, REG reduces to REG$_{0}$.

Case 5 occurs in situations where the researcher would like to test for the
independence between Gaussian vectors $x_{t}\in\mathbb{R}^{p}$ and $y_{t}%
\in\mathbb{R}^{n_{1}},$ given zero mean observations with $t=1,...,n_{1}%
+n_{2}.$ Partition the population and sample covariance matrices of the
observations $\left(  x_{t}^{\prime},y_{t}^{\prime}\right)  ^{\prime}$ as%
\[
\left(
\begin{array}
[c]{cc}%
\Sigma_{xx} & \Sigma_{xy}\\
\Sigma_{yx} & \Sigma_{yy}%
\end{array}
\right)  \text{ and }\left(
\begin{array}
[c]{cc}%
S_{xx} & S_{xy}\\
S_{yx} & S_{yy}%
\end{array}
\right)  ,
\]
respectively. Under the null hypothesis, $\Sigma_{xy}=0.$ The alternative of
interest is
\[
\Sigma_{xy}=\sqrt{\frac{n_{1}\theta}{n_{1}\theta+n_{1}+n_{2}}}\psi
\varphi^{\prime},
\]
where the vectors of nuisance parameters $\psi\in\mathbb{R}^{p}$ and
$\varphi\in\mathbb{R}^{n_{1}}$ are normalized so that
\[
\left\Vert \Sigma_{xx}^{-1/2}\psi\right\Vert =\left\Vert \Sigma_{yy}%
^{-1/2}\varphi\right\Vert =1.
\]
The peculiar form of the expression under the square root is chosen so as to
simplify various expressions in the analysis that follow.

The test can be based on the sample canonical correlations $\lambda
_{1},...,\lambda_{p}$, which are solutions to (\ref{basic}) with
\[
H=S_{xy}S_{yy}^{-1}S_{yx}\text{ and }E=S_{xx}.
\]
We label Case 5 as the `canonical correlation analysis' (CCA) case.
Remarkably, the sample canonical correlations also solve
(\ref{basic beta form}) with different $H$ and $E,$ such that $E$ is a central
Wishart matrix and $H$ is a non-central Wishart matrix conditionally on a
random non-centrality parameter (for details, see Theorem 11.3.2 of Muirhead (1982)).

Finally, as discussed in the Introduction, we also consider Case 0, which we
label as the `symmetric matrix denoising' (SMD) case. Given a $p\times p$
matrix $X=\Phi+Z/\sqrt{p},$ where $Z$ is a noise matrix from the Gaussian
Orthogonal Ensemble (GOE), a researcher would like to make inference about a
symmetric rank-one \textquotedblleft signal\textquotedblright\ matrix
$\Phi=\mathbb{\psi}\theta\mathbb{\psi}^{\prime}$. Recall, that a symmetric
matrix $Z$ belongs to GOE if its diagonal and sub-diagonal entries are
independently distributed as%
\[
Z_{ii}\sim N\left(  0,2\right)  \text{ and }Z_{ij}\sim N\left(  0,1\right)
\text{ if }i>j\text{.}%
\]
The null and the alternative hypotheses are given by (\ref{Basic hypotheses}).
The nuisance $\mathbb{\psi}\in\mathbb{R}^{p}$ is normalized so that
$\left\Vert \mathbb{\psi}\right\Vert =1$. The problem remains invariant under
the multiplication of $X$ from the left by an orthogonal matrix, and from the
right by its transpose. A maximal invariant statistic consists of the
solutions $\lambda_{1},...,\lambda_{p}$ to (\ref{basic}) with $H=X$ and
$E=I_{p}.$ We consider tests based on $\lambda_{1},...,\lambda_{p}$.

The SMD case can be viewed as a degenerate version of all of the above cases.
For example, consider REG$_{0}$ with%
\[
\mathbb{E}Y_{1}^{\ast}=\sqrt{\left(  p/n_{1}\right)  ^{1/2}n_{1}\theta}%
\varphi\psi^{\prime},
\]
so that the original value of the spike $\theta$ (see equation (\ref{REG0mean}%
)) is scaled by $\left(  p/n_{1}\right)  ^{1/2}$. Suppose now that $n_{1}$
diverges to infinity while $p$ is held constant. Then, by a Central Limit
Theorem (CLT),%
\begin{equation}
\Sigma^{-1/2}H\Sigma^{-1/2}-I_{p}=Z/\sqrt{n_{1}}+\sqrt{p/n_{1}}\eta\theta
\eta^{\prime}+o_{\mathrm{P}}\left(  n_{1}^{-1/2}\right)  ,
\label{equality link}%
\end{equation}
where $Z$ belongs to GOE and $\eta=\Sigma^{-1/2}\psi$. On the other hand,
equation (\ref{basic}) is equivalent to%
\begin{equation}
\det\left(  \Sigma^{-1/2}H\Sigma^{-1/2}-\lambda I_{p}\right)  =0.
\label{modified equation 1}%
\end{equation}
Multiplying it by $\sqrt{n_{1}/p}$ and using (\ref{equality link}), we see
that equation (\ref{modified equation 1}) degenerates to%
\[
\det\left(  Z/\sqrt{p}+\eta\theta\eta^{\prime}-\mu I_{p}\right)  =0\text{ with
}\mu=\sqrt{n_{1}/p}\left(  \lambda-1\right)  .
\]
Hence, REG$_{0}$ degenerates to SMD.

For the reader's convenience, we summarize links between the different cases
and the definitions of the corresponding matrices $H$ and $E$ in Figure
\ref{diagram1}. We denote the $p$-dimensional Wishart distribution with $n$
degrees of freedom, covariance parameter $\Sigma$, and non-centrality
parameter $\Psi$ as $W_{p}\left(  n,\Sigma,\Psi\right)  .$ Recall that, if
$A=B^{\prime}B,$ where the $n\times p$ matrix $B$ is $N\left(  M,I_{n}%
\otimes\Sigma\right)  ,$ then $A\sim W_{p}\left(  n,\Sigma,\Psi\right)  $ with
the non-centrality $\Psi=\Sigma^{-1}M^{\prime}M$. Notation $W_{p}\left(
n,\Sigma\right)  $ is used for the central Wishart distribution. Without loss
of generality, we assume that $\Sigma=I_{p}$.

All the cases eventually degenerate to SMD via sequential asymptotic links.
Cases SMD, PCA, and REG$_{0}$, forming the upper half of the diagram,
correspond to random $H$ and deterministic $E.$ The cases in the lower half of
the diagram correspond to both $H$ and $E$ being random. Cases PCA and SigD
are \textquotedblleft parallel\textquotedblright\ to cases REG$_{0}$ and REG
in the sense that the alternative hypothesis is characterized by a rank one
perturbation of the covariance and of the non-centrality parameter of $H$ for
the former and for the latter two cases, respectively. Case CCA
\textquotedblleft stands alone\textquotedblright\ because of the different
structure of $H$ and $E.$ As discussed above, CCA can be reinterpreted in
terms of $H$ and $E$ such that $E$ is Wishart, but $H$ is a non-central
Wishart only after conditioning on a random non-centrality parameter.%

%TCIMACRO{\FRAME{ftbpFU}{4.4702in}{4.5238in}{0pt}{\Qcb{Matrices $H$ and $E,$
%and links between the different cases. Matrix $\Phi$ has the form $\psi
%\theta\psi^{\prime}$ with $\theta\geq0$ and $\left\Vert \psi\right\Vert =1$.}%
%}{\Qlb{diagram1}}{diagram1.eps}{\special{ language "Scientific Word";
%type "GRAPHIC";  maintain-aspect-ratio TRUE;  display "USEDEF";
%valid_file "F";  width 4.4702in;  height 4.5238in;  depth 0pt;
%original-width 7.3864in;  original-height 7.4737in;  cropleft "0";
%croptop "1";  cropright "1";  cropbottom "0";
%filename 'diagram1.eps';file-properties "XNPEU";}} }%
%BeginExpansion
\begin{figure}[ptb]%
\centering
\includegraphics[
height=4.5238in,
width=4.4702in
]%
{diagram1.eps}%
\caption{Matrices $H$ and $E,$ and links between the different cases. Matrix
$\Phi$ has the form $\psi\theta\psi^{\prime}$ with $\theta\geq0$ and
$\left\Vert \psi\right\Vert =1$.}%
\label{diagram1}%
\end{figure}
%EndExpansion


\section{The likelihood ratios}

Our goal is to study the asymptotic behavior of the likelihood ratios, which
are defined as the ratios of the joint density of $\lambda_{1},...,\lambda
_{p}$ under the alternative to that under the null hypothesis, where both
densities are evaluated at the observed values of the $\lambda$'s. Let
\[
\Lambda=\operatorname*{diag}\left\{  \lambda_{1},...,\lambda_{p}\right\}  ,
\]
and let us denote the likelihood ratio corresponding to particular case `Case'
= `SMD', `PCA', etc. as $L^{(Case)}\left(  \theta;\Lambda\right)  .$ Then%
\begin{equation}
L^{(Case)}\left(  \theta;\Lambda\right)  =\alpha\left(  \theta\right)  \left.
_{\bar{p}}F_{\bar{q}}\left(  a,b;\Psi,\Lambda\right)  \right.  ,
\label{LRgeneral}%
\end{equation}
where $\Psi$ is a $p$-dimensional matrix $\operatorname*{diag}\left\{
\Psi_{11},0,...,0\right\}  ,$ and the values of $\Psi_{11},$ $\alpha\left(
\theta\right)  ,$ $\bar{p},$ $\bar{q},$ $a,$ and $b$ are as given in Table
\ref{Table 2}.%

%TCIMACRO{\TeXButton{B}{\begin{table}[tbp] \centering}}%
%BeginExpansion
\begin{table}[tbp] \centering
%EndExpansion%
\begin{tabular}
[c]{llllll}%
Case & $_{\bar{p}}F_{\bar{q}}$ & $\alpha\left(  \theta\right)  $ & $a$ & $b$ &
$\Psi_{11}$\\\hline
SMD & $_{0}F_{0}$ & $\exp\left(  -p\theta^{2}/4\right)  $ & \_ & \_ & $\theta
p/2\bigskip$\\
PCA & $_{0}F_{0}$ & $\left(  1+\theta\right)  ^{-n_{1}/2}$ & \_ & \_ & $\theta
n_{1}/(2\left(  1+\theta\right)  )\bigskip$\\
SigD & $_{1}F_{0}$ & $\left(  1+\theta\right)  ^{-n_{1}/2}$ & $n/2$ & \_ &
$\theta n_{1}/\left(  n_{2}\left(  1+\theta\right)  \right)  \bigskip$\\
REG$_{0}$ & $_{0}F_{1}$ & $\exp\left(  -n_{1}\theta/2\right)  $ & \_ &
$n_{1}/2$ & $\theta n_{1}^{2}/4\bigskip$\\
REG & $_{1}F_{1}$ & $\exp\left(  -n_{1}\theta/2\right)  $ & $n/2$ & $n_{1}/2$
& $\theta n_{1}^{2}/\left(  2n_{2}\right)  \bigskip$\\
CCA & $_{2}F_{1}$ & $\left(  1+n_{1}\theta/n\right)  ^{-n/2}$ & $\left(
n/2,n/2\right)  $ & $n_{1}/2$ & $\theta n_{1}^{2}/\left(  n_{2}^{2}+n_{2}%
n_{1}\left(  1+\theta\right)  \right)  $\\\hline
\end{tabular}
\caption{Parameters of the explicit expression (\ref{LRgeneral}) for the likelihood ratios. Here $n\equiv n_1+n_2$.}\label{Table 2}%
%TCIMACRO{\TeXButton{E}{\end{table}}}%
%BeginExpansion
\end{table}%
%EndExpansion


We prove that $L^{(SMD)}\left(  \theta;\Lambda\right)  $ is as in
(\ref{LRgeneral}) in the Appendix. For PCA, the explicit form of the
likelihood ratio is derived in Onatski et al (2013). For SigD, REG$_{0}$, and
REG, the expressions (\ref{LRgeneral}) with the parameters given in Table
\ref{Table 2} follow, respectively, from equations (65), (68), and (73) of
James (1964). For CCA, the expression is a corollary of Theorem 11.3.2 of
Muirhead (1982).

Recall that hypergeometric functions of two matrix arguments $\Psi$ and
$\Lambda$ are defined as%
\[
_{\bar{p}}F_{\bar{q}}\left(  a,b;\Psi,\Lambda\right)  =\sum_{k=0}^{\infty
}\frac{1}{k!}\sum_{\kappa\vdash k}\frac{\left(  a_{1}\right)  _{\kappa
}...\left(  a_{\bar{p}}\right)  _{\kappa}}{\left(  b_{1}\right)  _{\kappa
}...\left(  b_{\bar{q}}\right)  _{\kappa}}\frac{C_{\kappa}\left(  \Psi\right)
C_{\kappa}\left(  \Lambda\right)  }{C_{\kappa}\left(  I_{p}\right)  },
\]
where $a=\left(  a_{1},...,a_{\bar{p}}\right)  $ and $b=\left(  b_{1}%
,....,b_{\bar{q}}\right)  $ are parameters, $\kappa$ are partitions of the
integer $k$, $\left(  a_{j}\right)  _{\kappa}$ and $\left(  b_{i}\right)
_{\kappa}$ are the generalized Pochhammer symbols, and $C_{\kappa}$ are the
zonal polynomials (see Muirhead (1982), Definition 7.3.2). As mentioned in the
Introduction, James' (1964) classification of the multivariate statistical
problems is based on the type of $_{\bar{p}}F_{\bar{q}}$ that occur in related
probability distributions. The function $_{0}F_{0}$ of \textit{exponential
type} corresponds to the first class represented by PCA; the function
$_{1}F_{0}$ of \textit{binomial type} corresponds to the second class
represented by SigD; the function $_{0}F_{1}$ of \textit{Bessel type} is
associated with the third class represented by REG$_{0}$; the
\textit{confluent hypergeometric function} $_{1}F_{1}$ is associated with the
fourth class represented by REG; and the \textit{Gaussian hypergeometric
function} $_{2}F_{1}$ corresponds to the fifth class represented by CCA. Note
that some links between the cases illustrated in Figure \ref{diagram1} can
also be established via asymptotic relations between the hypergeometric
functions in the different rows of Table \ref{Table 2}. For example, the links
REG$\mapsto$REG$_{0}$ and SigD$\mapsto$PCA as $n_{2}\rightarrow\infty$ while
$p$ and $n_{1}$ are held constant follow from the confluence relations (see,
for example, chapter 3.5 of Luke (1969))
\begin{align*}
\left.  _{0}F_{1}\left(  b;\Psi,\Lambda\right)  \right.   &  =\lim
_{a\rightarrow\infty}\left.  _{1}F_{1}\left(  a,b;a^{-1}\Psi,\Lambda\right)
\right.  \text{ and}\\
\left.  _{0}F_{0}\left(  \Psi,\Lambda\right)  \right.   &  =\lim
_{a\rightarrow\infty}\left.  _{1}F_{0}\left(  a;a^{-1}\Psi,\Lambda\right)
\right.  .
\end{align*}


In the next section, we shall study the asymptotic behavior of the likelihood
ratios (\ref{LRgeneral}) as $n_{1},n_{2},$ and $p$ go to infinity so that%
\begin{equation}
c_{1}\equiv\frac{p}{n_{1}}\rightarrow\gamma_{1}\in\left(  0,1\right)  \text{
and }c_{2}\equiv\frac{p}{n_{2}}\rightarrow\gamma_{2}\in\left(  0,1\right]  .
\label{asymptotic regime}%
\end{equation}
We denote this asymptotic regime as $\mathbf{n},p\rightarrow_{\mathbf{\gamma}%
}\infty,$ where $\mathbf{n}=\left\{  n_{1},n_{2}\right\}  $ and
$\mathbf{\gamma}=\left\{  \gamma_{1},\gamma_{2}\right\}  .$ To make our
exposition as uniform as possible, we use this notation for all the cases,
even though the simpler ones, such as SMD, do not refer to $\mathbf{n}$. In
the Conclusion, we briefly discuss possible extensions of our analysis to the
situations with $\gamma_{1}\geq1$.

We are interested in the asymptotics of the likelihood ratios under the null
hypothesis, that is when the true value of the spike, $\theta_{0}$, equals
zero. Before turning to the next section, let us provide a relevant background
on the asymptotics of $\Lambda$. Under the null, $\lambda_{1},...,\lambda_{p}$
are the eigenvalues of $GOE/\sqrt{p}$ in SMD case; of $W_{p}\left(
n_{1},I_{p}\right)  /n_{1}$ in PCA and REG$_{0}$ cases; and of a scaled (by a
factor of $n_{2}/n_{1}$) $p$-dimensional multivariate beta matrix with
parameters $n_{1}/2$ and $n_{2}/2$ in SigD, REG, and CCA cases. For a
definition of the multivariate beta, see Muirhead (1982), p. 110.

Let%
\[
\hat{F}^{(Case)}\left(  \lambda\right)  =\frac{1}{p}%
%TCIMACRO{\dsum _{j=1}^{p}}%
%BeginExpansion
{\displaystyle\sum_{j=1}^{p}}
%EndExpansion
1\left\{  \lambda_{j}\leq\lambda\right\}
\]
be the empirical distribution of $\lambda_{1},...,\lambda_{p}$. As is well
known (see Bai (1999)), as $\mathbf{n},p\rightarrow_{\mathbf{\gamma}}\infty,$
$\hat{F}^{(Case)}$ almost surely (a.s.) weakly converges
\[
\hat{F}^{(Case)}\Rightarrow F_{\mathbf{\gamma}}^{\lim},
\]
where $F_{\mathbf{\gamma}}^{\lim}$ is the semi-circle distribution $F^{SC}$ in
SMD case; the Marchenko-Pastur distribution $F_{\gamma_{1}}^{MP}$ in PCA and
REG$_{0}$ cases; and the (scaled) Wachter distribution $F_{\mathbf{\gamma}%
}^{W}$ in SigD, REG, and CCA cases. Table \ref{Table 3} reports the explicit
forms of these limiting distributions. Note that the cumulative distribution
functions $F_{\mathbf{\gamma}}^{\lim}\left(  \lambda\right)  $ are linked in
the sense that $F_{\mathbf{\gamma}}^{W}\left(  \lambda\right)  \rightarrow
F_{\gamma_{1}}^{MP}\left(  \lambda\right)  $ when $\gamma_{2}\rightarrow0$ and
$F_{\gamma_{1}}^{MP}\left(  \sqrt{\gamma_{1}}\lambda+1\right)  \rightarrow
F^{SC}\left(  \lambda\right)  $ when $\gamma_{1}\rightarrow0$.%

%TCIMACRO{\TeXButton{B}{\begin{table}[tbp] \centering}}%
%BeginExpansion
\begin{table}[tbp] \centering
%EndExpansion%
\begin{tabular}
[c]{lllll}%
Case & $F_{\mathbf{\gamma}}^{\lim}$ & density for $\beta_{-}\leq\lambda
\leq\beta_{+}$ & $\beta_{\pm}$ & Threshold $\bar{\theta}$\\\hline
SMD & SC & $\frac{1}{2\pi}\sqrt{\left(  \beta_{+}-\lambda\right)  \left(
\lambda-\beta_{-}\right)  }$ & $\pm2$ & $1\bigskip$\\
$%
\begin{array}
[c]{l}%
\text{PCA}\\
\text{REG}_{0}%
\end{array}
$ & MP & $\frac{1}{2\pi\gamma_{1}\lambda}\sqrt{\left(  \beta_{+}%
-\lambda\right)  \left(  \lambda-\beta_{-}\right)  }$ & $\left(  1\pm
\sqrt{\gamma_{1}}\right)  ^{2}$ & $\sqrt{\gamma_{1}}\bigskip$\\
$%
\begin{array}
[c]{l}%
\text{SigD}\\
\text{REG}\\
\text{CCA}%
\end{array}
$ & W & $\frac{\gamma_{1}+\gamma_{2}}{2\pi\gamma_{1}\lambda\left(  \gamma
_{1}-\gamma_{2}\lambda\right)  }\sqrt{\left(  \beta_{+}-\lambda\right)
\left(  \lambda-\beta_{-}\right)  }$ & $\gamma_{1}\left(  \frac{\rho\pm1}%
{\rho\pm\gamma_{2}}\right)  ^{2}$ & $\frac{\gamma_{2}+\rho}{1-\gamma_{2}}%
$\\\hline
\end{tabular}
\caption{The semi-circle, Marchenko-Pastur, and (scaled) Wachter distributions. Here $\rho=\sqrt{\gamma_1+\gamma_2-\gamma_{1}\gamma_{2}}$. In the case where $\gamma_1>1$, which is not
considered in this paper, the Marchenko-Pastur and Wachter distributions will also have mass $(\gamma_1-1)/\gamma_1$ at zero. Column `Threshold $\bar{\theta}$' reports the values of the phase transition thresholds.}\label{Table 3}%
%TCIMACRO{\TeXButton{E}{\end{table}}}%
%BeginExpansion
\end{table}%
%EndExpansion


For what follows it will be important that the centered \textit{linear
spectral statistics}
\begin{equation}%
%TCIMACRO{\dsum _{j=1}^{p}}%
%BeginExpansion
{\displaystyle\sum_{j=1}^{p}}
%EndExpansion
\varphi\left(  \lambda_{j}\right)  -p\int\varphi\left(  \lambda\right)
\mathrm{d}F_{\mathbf{c}}^{\lim}\left(  \lambda\right)  ,
\label{linear spectral statistics}%
\end{equation}
where $\varphi$ is a `well-behaved' function, converge in distribution to
Gaussian random variables. The corresponding CLTs are established in Bai and
Yao (2005), Bai and Silverstein (2004), and Zheng (2012) for the cases of the
semi-circle, Marchenko-Pastur, and Wachter limiting distributions,
respectively. Note that the centering constant is defined in terms of
$F_{\mathbf{c}}^{\lim},$ where $\mathbf{c}=\left\{  c_{1},c_{2}\right\}  .$
That is, the \textquotedblleft correct centering\textquotedblright\ can be
computed using the densities from Table \ref{Table 3}, where $\gamma_{1}$ and
$\gamma_{2}$ are replaced by $c_{1}\equiv p/n_{1}$ and $c_{2}\equiv p/n_{2}$, respectively.

Finally, let us note the behavior of the largest eigenvalue $\lambda_{1}$
under the alternative hypothesis. As is well known, $\lambda_{1}$ a.s.
converges to the upper boundary of support of $F_{\mathbf{\gamma}}^{\lim}$ as
long as $\theta$ remains below the phase transition threshold $\bar{\theta}$.
The value of the threshold is reported in the last column of Table
\ref{Table 3}. When $\theta>\bar{\theta}$, $\lambda_{1}$ separates from `the
bulk' of the other eigenvalues and a.s. converges to a point strictly above
the upper boundary of the support of $F_{\mathbf{\gamma}}^{\lim}.$ For
details, we refer the reader to Ma\"{\i}da (2007), Baik and Silverstein
(2006), Nadakuditi and Silverstein (2010), Onatski (2007), Dharmawansa et al
(2014a), and Bao et al (2014) for cases SMD, PCA, SigD, REG$_{0}$, REG, and
CCA, respectively.

The fact that $\lambda_{1}$ converges to different limits under the null and
under the alternative hypothesis sheds light on the behavior of the likelihood
ratio when $\theta$ is above the phase transition threshold. In such cases,
which can be called the cases of super-critical $\theta$, the likelihood ratio
degenerates. The sequences of measures corresponding to the distributions of
$\Lambda$ under the null and under super-critical alternatives are
asymptotically mutually singular as $\mathbf{n},p\rightarrow_{\mathbf{\gamma}%
}\infty$ (see Montanari et al (2014) and Onatski et al (2013) for a detailed
analysis of SMD and PCA cases). In contrast, as we shall show below, the
sequences of measures corresponding to the distributions of $\Lambda$ under
the null and under sub-critical alternatives ($\theta$ is below the threshold)
are mutually contiguous, and the likelihood ratio converges to a Gaussian process.

\section{Contour integral representation}

Asymptotic behavior of the likelihood ratios (\ref{LRgeneral}) depends on that
of $_{\bar{p}}F_{\bar{q}}\left(  a,b;\Psi,\Lambda\right)  $. There is a large
and well established literature on the asymptotics of $_{\bar{p}}F_{\bar{q}%
}\left(  a,b;\Psi,\Lambda\right)  $ when the parameters and the norm of the
matrix arguments grow while the dimensionality of the latter remains fixed
(see Muirhead (1978) for a review). In contrast, relatively little is known
about the asymptotic regime that allows the dimensionality of the matrix
arguments $\Psi,\Lambda$ diverge to infinity. In this paper, we investigate
such an asymptotic regime. We exploit the fact that, since we study
single-spiked models, the matrix argument $\Psi$ has rank one. This allows us
to represent $_{\bar{p}}F_{\bar{q}}\left(  a,b;\Psi,\Lambda\right)  $ in the
form of a contour integral of a hypergeometric function with a single scalar
argument. Such a representation implies contour integral representations for
the corresponding likelihood ratios, which we summarize in the following
lemma. The results of the lemma are used below to derive the asymptotics of
the likelihood ratios via the Laplace approximation.

In what follows, we omit the superscripts `(Case)' and `lim' for quantities
such as $L^{(Case)}\left(  \theta;\Lambda\right)  ,$ $\hat{F}^{(Case)}\left(
\lambda\right)  ,$ and $F_{\mathbf{c}}^{\lim}\left(  \lambda\right)  $ to
simplify our notation. However, we shall use these superscripts to identify
particular instances, when necessary.

\begin{lemma}
\label{DJLemma}Assume that $p\leq\min\left\{  n_{1},n_{2}\right\}  .$ Let
$\mathcal{K}$ be a contour in the complex plane $\mathbb{C}$ that starts at
$-\infty$, encircles $0$ and $\lambda_{1},...,\lambda_{p}$ counterclockwise,
and returns to $-\infty$. Then%
\begin{equation}
L\left(  \theta;\Lambda\right)  =\frac{\Gamma\left(  s+1\right)  \alpha\left(
\theta\right)  q_{s}}{\Psi_{11}^{s}2\pi\mathrm{i}}\int_{\mathcal{K}}\left.
_{\bar{p}}F_{\bar{q}}\right.  \left(  a-s,b-s;\Psi_{11}z\right)
%TCIMACRO{\dprod _{j=1}^{p}}%
%BeginExpansion
{\displaystyle\prod_{j=1}^{p}}
%EndExpansion
\left(  z-\lambda_{j}\right)  ^{-1/2}\mathrm{d}z, \label{integral}%
\end{equation}
where $s=p/2-1,$ the values of $\alpha\left(  \theta\right)  ,$ $\Psi_{11},$
$a,$ $b,$ $\bar{p},$ and $\bar{q}$ for the different cases are given in Table
\ref{Table 2}; $a-s$ and $b-s$ denote vectors with elements $a_{j}-s$ and
$b_{j}-s,$ respectively; the hypergeometric function under the integral is the
standard hypergeometric function of a scalar argument; and
\[
q_{s}=%
%TCIMACRO{\dprod _{j=1}^{\bar{p}}}%
%BeginExpansion
{\displaystyle\prod_{j=1}^{\bar{p}}}
%EndExpansion
\frac{\Gamma\left(  a_{j}-s\right)  }{\Gamma\left(  a_{j}\right)  }%
%TCIMACRO{\dprod _{i=1}^{\bar{q}}}%
%BeginExpansion
{\displaystyle\prod_{i=1}^{\bar{q}}}
%EndExpansion
\frac{\Gamma\left(  b_{i}\right)  }{\Gamma\left(  b_{i}-s\right)  }.
\]
In cases SigD and CCA, we require, in addition, that the contour $\mathcal{K}$
does not intersect $\left[  \Psi_{11}^{-1},\infty\right)  $, which ensures the
analyticity of the integrand in an open subset of $\mathbb{C}$ that includes
$\mathcal{K}$.
\end{lemma}

The statement of the lemma immediately follows from Proposition 1 of
Dharmawansa and Johnstone (2014) and from equation (\ref{LRgeneral}). Our next
step is to apply the Laplace approximation to integrals (\ref{integral}). To
this end, we shall transform the right hand side of (\ref{integral}) so that
it has a \textquotedblleft Laplace form\textquotedblright%
\begin{equation}
L\left(  \theta;\Lambda\right)  =\sqrt{\pi p}\frac{1}{2\pi\mathrm{i}}%
\int_{\mathcal{K}}\exp\left\{  -pf(z)\right\}  g(z)\mathrm{d}z.
\label{Laplace form}%
\end{equation}
Leaving $\sqrt{\pi p}/\left(  2\pi\mathrm{i}\right)  $ separate from $g(z)$
allows us to choose $f(z)$ and $g(z)$ that are bounded in probability, and
makes some of the expressions below more compact. In order to apply the
Laplace approximation, we shall deform the contour of integration so that it
passes through a critical point $z_{0}$ of $f(z)$ and is such that
$\operatorname{Re}f(z)$ is strictly increasing as $z$ moves away from $z_{0}$
along the contour, at least in a vicinity of $z_{0}$.

\subsection{The Laplace form}

We shall transform (\ref{integral}) to (\ref{Laplace form}) in three steps. As
a result, functions $f$ and $g$ will have the forms of a sum and a product,%
\begin{align*}
f\left(  z\right)   &  =f_{I}+f_{II}\left(  z\right)  +f_{III}\left(
z\right)  \text{ and}\\
g(z)  &  =g_{I}\times g_{II}\left(  z\right)  \times g_{III}(z),
\end{align*}
where $f_{I}$ and $g_{I}$ do not depend on $z$.

First, using the definitions of $\alpha\left(  \theta\right)  ,$ $q_{s},$
$\Psi_{11}$ and employing Stirling's approximation, we obtain a decomposition%
\begin{equation}
\frac{\Gamma\left(  s+1\right)  \alpha\left(  \theta\right)  q_{s}}{\sqrt{\pi
p}\Psi_{11}^{s}}=\exp\left\{  -pf_{I}\right\}  g_{I}, \label{decomposition 0}%
\end{equation}
where $g_{I}$ remains boudned as $\mathbf{n},p\rightarrow_{\mathbf{\gamma}%
}\infty$. The values of $2f_{I}$ and $g_{I}$ are given in Table \ref{Table 2a}%
. It should be noted that $f_{I}^{(REG)},f_{I}^{(CCA)}\rightarrow
f_{I}^{(REG_{0})}$ and $f_{I}^{(SigD)}\rightarrow f_{I}^{(PCA)}$ as
$c_{2}\rightarrow0$.%

%TCIMACRO{\TeXButton{B}{\begin{table}[tbp] \centering}}%
%BeginExpansion
\begin{table}[tbp] \centering
%EndExpansion%
\begin{tabular}
[c]{lll}%
Case & $2f_{I}$ & $g_{I}/(1+o(1))$\\\hline
SMD & $\medskip1+\theta^{2}/2+\ln\theta$ & $\theta\bigskip$\\
PCA & $1+\frac{1-c_{1}}{c_{1}}\ln\left(  1+\theta\right)  +\ln\frac{\theta
}{c_{1}}$ & $\theta c_{1}^{-1}\left(  1+\theta\right)  ^{-1}\bigskip$\\
SigD & $2f_{I}^{(PCA)}-1+\ln\frac{c_{1}+c_{2}}{c_{1}}-\frac{r^{2}}{c_{1}c_{2}%
}\ln\frac{r^{2}}{c_{1}+c_{2}}$ & $\theta c_{1}^{-2}\left(  1+\theta\right)
^{-1}r\left(  c_{1}+c_{2}\right)  ^{1/2}\bigskip$\\
REG$_{0}$ & $1+\frac{\theta+c_{1}}{c_{1}}+\ln\frac{\theta}{c_{1}}%
+\frac{1-c_{1}}{c_{1}}\ln\left(  1-c_{1}\right)  $ & $\theta c_{1}^{-1}\left(
1-c_{1}\right)  ^{-1/2}\bigskip$\\
REG & $2f_{I}^{(REG_{0})}-1+\ln\frac{c_{1}+c_{2}}{c_{1}}-\frac{r^{2}}%
{c_{1}c_{2}}\ln\frac{r^{2}}{c_{1}+c_{2}}$ & $\theta c_{1}^{-2}\left(
1-c_{1}\right)  ^{-1/2}r\left(  c_{1}+c_{2}\right)  ^{1/2}\bigskip$\\
CCA & $2f_{I}^{(REG)}-1-\frac{\theta}{c_{1}}-\frac{r^{2}}{c_{1}c_{2}}\ln
\frac{r^{2}}{c_{1}l}$ & $\theta c_{1}^{-3}\left(  1-c_{1}\right)  ^{-1/2}%
r^{2}\left(  c_{1}+c_{2}\right)  l^{-1}\medskip$\\\hline
\end{tabular}
\caption{Values of $2f_{I}$ and $g_{I}/(1+o(1))$ for the different cases.
The terms $o(1)$ do not depend on $\theta$ and converge to zero as
$\mathbf{n},p\rightarrow _{\mathbf{\gamma }}\infty $. The term $r^{2}$ is defined as
$r^{2}=c_{1}+c_{2}-c_{1}c_{2}$. The term $l \equiv l(\theta)$ is defined as $l(\theta)=1+(1+\theta)c_2/c_1$.}\label{Table 2a}%
%TCIMACRO{\TeXButton{E}{\end{table}}}%
%BeginExpansion
\end{table}%
%EndExpansion


Next, we consider the decomposition%
\begin{equation}%
%TCIMACRO{\dprod _{j=1}^{p}}%
%BeginExpansion
{\displaystyle\prod_{j=1}^{p}}
%EndExpansion
\left(  z-\lambda_{j}\right)  ^{-1/2}=\exp\left\{  -pf_{II}(z)\right\}
g_{II}(z), \label{step1}%
\end{equation}
where%
\begin{equation}
2f_{II}(z)=\int\ln\left(  z-\lambda\right)  \mathrm{d}F_{\mathbf{c}}(\lambda),
\label{f1Case}%
\end{equation}
and%
\begin{equation}
g_{II}(z)=\exp\left\{  -\frac{p}{2}\int\ln\left(  z-\lambda\right)
\mathrm{d}\left(  \hat{F}\left(  \lambda\right)  -F_{\mathbf{c}}\left(
\lambda\right)  \right)  \right\}  . \label{Deltap}%
\end{equation}
For $f_{II}(z)$ and $g_{II}(z)$ to be well-defined we need $z$ not to belong
to the support of $F_{\mathbf{c}},$ which we assume.\footnote{By definition,
contour $\mathcal{K}$ encircles the support of $\hat{F},$ and hence
$z\in\mathcal{K}$ does not belong to such support.} Note that $g_{II}(z)$ is
the exponent of a linear spectral statistic, which converges to a Gaussian
random variable as $\mathbf{n},p\rightarrow_{\mathbf{\gamma}}\infty$ under the
null hypothesis. Since $F_{\mathbf{c}}^{W}(\lambda)\rightarrow F_{c_{1}}%
^{MP}(\lambda)$ as $c_{2}\rightarrow0,$ we have $f_{II}^{(SigD)}%
(z)=f_{II}^{(REG)}(z)=f_{II}^{(CCA)}(z)$ converging to $f_{II}^{(PCA)}%
(z)=f_{II}^{(REG_{0})}(z)$.

Finally, we obtain a decomposition%
\begin{equation}
\left.  _{\bar{p}}F_{\bar{q}}\right.  \left(  a-s,b-s;\Psi_{11}z\right)
=\exp\left\{  -pf_{III}(z)\right\}  g_{III}(z). \label{LaplaceForm}%
\end{equation}
For SMD, PCA, and SigD, the corresponding $\left.  _{\bar{p}}F_{\bar{q}%
}\right.  $ can be expressed in terms of elementary functions, and we set%
\begin{equation}
2f_{III}(z)=\left\{
\begin{array}
[c]{ll}%
-z\theta & \text{for SMD}\\
-z\theta/\left(  c_{1}\left(  1+\theta\right)  \right)  \text{ } & \text{for
PCA}\\
\ln\left[  1-c_{2}z\theta/\left\{  c_{1}\left(  1+\theta\right)  \right\}
\right]  r^{2}/\left(  c_{1}c_{2}\right)  & \text{for SigD}%
\end{array}
\right.  , \label{elementary}%
\end{equation}
and%
\begin{equation}
g_{III}(z)=\left\{
\begin{array}
[c]{ll}%
1 & \text{for SMD and PCA}\\
\left[  1-c_{2}z\theta/\left\{  c_{1}\left(  1+\theta\right)  \right\}
\right]  ^{-1} & \text{for SigD}%
\end{array}
\right.  . \label{component2SMD}%
\end{equation}
As $c_{2}\rightarrow0$, $f_{III}^{(SigD)}(z)$ converges to $f_{III}%
^{(PCA)}(z).$ Since, as has been shown above, a similar convergence holds for
$f_{I}$ and $f_{II},$ we have $f^{(SigD)}(z)\rightarrow f^{(PCA)}(z)$ as
$c_{2}\rightarrow0$. Combining (\ref{f1Case}) and (\ref{elementary}) with the
information supplied by Table \ref{Table 2a}, we also see that $f^{(PCA)}%
(z)\rightarrow f^{(SMD)}(z)$ as $c_{1}\rightarrow0$ after the transformations
$\theta\mapsto\sqrt{c_{1}}\theta$ and $z\mapsto\sqrt{c_{1}}z+1$.

Unfortunately, for REG$_{0},$ REG, and CCA, the corresponding $\left.
_{\bar{p}}F_{\bar{q}}\right.  $ do not admit exact representations in terms of
elementary functions. Therefore, we shall consider their asymptotic
approximations instead. Let%
\[
m=\left(  n_{1}-p\right)  /2\text{ and }\varepsilon=\left(  n-p\right)
/\left(  n_{1}-p\right)  .
\]
Further, let%
\begin{equation}
\eta_{j}=\left\{
\begin{array}
[c]{ll}%
z\theta/\left(  1-c_{1}\right)  ^{2} & \text{for }j=0\\
z\theta c_{2}/\left[  c_{1}\left(  1-c_{1}\right)  \right]  & \text{for }j=1\\
z\theta c_{2}^{2}/\left[  c_{1}^{2}l\left(  \theta\right)  \right]  &
\text{for }j=2
\end{array}
\right.  , \label{etas}%
\end{equation}
where%
\begin{equation}
l\left(  \theta\right)  =1+\left(  1+\theta\right)  c_{2}/c_{1}.
\label{loftheta}%
\end{equation}
With this notation, we have%
\begin{equation}
\left.  _{\bar{p}}F_{\bar{q}}\right.  =\left\{
\begin{array}
[c]{ll}%
\left.  _{0}F_{1}\right.  \left(  m+1;m^{2}\eta_{0}\right)  \equiv F_{0} &
\text{for REG}_{0}\\
\left.  _{1}F_{1}\right.  \left(  m\varepsilon+1,m+1;m\eta_{1}\right)  \equiv
F_{1} & \text{for REG}\\
\left.  _{2}F_{1}\right.  \left(  m\varepsilon+1,m\varepsilon+1;m+1;\eta
_{2}\right)  \equiv F_{2} & \text{for CCA}%
\end{array}
\right.  . \label{pFq}%
\end{equation}


The function $F_{0}$ can be expressed in terms of the modified Bessel function
of the first kind $I_{m}\left(  \cdot\right)  $ as (see Abramowitz and Stegun
(1964), equation 9.6.47)%
\begin{equation}
F_{0}=\Gamma\left(  m+1\right)  \left(  m^{2}\eta_{0}\right)  ^{-m/2}%
I_{m}\left(  2m\eta_{0}^{1/2}\right)  .\label{bessel rep}%
\end{equation}
This representation allows us to use a known uniform asymptotic approximation
of the Bessel function (see Abramowitz and Stegun (1964), equation 9.7.7) to
obtain the following lemma. Let%
\begin{equation}
\varphi_{0}\left(  t\right)  =\ln t-t-\eta_{0}/t+1\text{ and }t_{0}=\left(
1+\sqrt{1+4\eta_{0}}\right)  /2.\label{phi0}%
\end{equation}
Further, for any $\delta>0,$ let $\Omega_{0\delta}$ be the set of $\eta_{0}%
\in\mathbb{C}$ such that%
\[
\left\vert \arg\eta_{0}\right\vert \leq\pi-\delta,\text{ and }\eta_{0}\neq0.
\]


\begin{lemma}
\label{uniform1}As $m\rightarrow\infty,$ we have%
\begin{equation}
F_{0}=\left(  1+4\eta_{0}\right)  ^{-1/4}\exp\left\{  -m\varphi_{0}\left(
t_{0}\right)  \right\}  \left(  1+o(1)\right)  . \label{BesselApprox}%
\end{equation}
The convergence $o(1)\rightarrow0$ holds uniformly with respect to $\eta
_{0}\in\Omega_{0\delta}$ for any $\delta>0$.
\end{lemma}

We would like to point out that the right hand side of (\ref{BesselApprox})
can be formally linked, via (\ref{bessel rep}), to the saddle-point
approximation of the integral representation (see Watson (1944), p. 181)%
\[
I_{m}\left(  2m\eta_{0}^{1/2}\right)  =\frac{\eta_{0}^{m/2}e^{m}}%
{2\pi\mathrm{i}}\int_{-\infty}^{(0+)}\exp\left\{  -m\varphi_{0}\left(
t\right)  \right\}  t^{-1}\mathrm{d}t.
\]
Point $t_{0}$ can be interpreted as a saddle point of $\varphi_{0}\left(
t\right)  ,$ and the term $\left(  1+4\eta_{0}\right)  ^{-1/4}$ in
(\ref{BesselApprox}) can be interpreted as a factor of $\left(  \varphi
_{0}^{\prime\prime}\left(  t_{0}\right)  \right)  ^{-1/2}$.

To obtain uniform asymptotic approximations to functions $F_{1}$ and $F_{2}$,
we use the contour integral representations (see Olver et al (2010), equations
13.4.9 and 15.6.2)
\begin{equation}
F_{j}=\frac{C_{m}}{2\pi\mathrm{i}}\int_{0}^{(1+)}\exp\left\{  -m\varphi
_{j}\left(  t\right)  \right\}  \psi_{j}\left(  t\right)  \mathrm{d}t,
\label{integralrepF11}%
\end{equation}
where%
\begin{equation}
C_{m}=\frac{\Gamma\left(  m+1\right)  \Gamma\left(  m\left(  \varepsilon
-1\right)  +1\right)  }{\Gamma\left(  m\varepsilon+1\right)  }, \label{Calpha}%
\end{equation}%
\begin{equation}
\varphi_{j}(t)=\left\{
\begin{array}
[c]{ll}%
-\eta_{j}t-\varepsilon\ln t+\left(  \varepsilon-1\right)  \ln\left(
t-1\right)  & \text{for }j=1\\
-\varepsilon\ln\left(  t/\left(  1-\eta_{j}t\right)  \right)  +\left(
\varepsilon-1\right)  \ln\left(  t-1\right)  & \text{for }j=2
\end{array}
\right.  , \label{phij}%
\end{equation}
and%
\begin{equation}
\psi_{j}\left(  t\right)  =\left\{
\begin{array}
[c]{ll}%
\left(  t-1\right)  ^{-1} & \text{for }j=1\\
\left(  t-1\right)  ^{-1}\left(  1-\eta_{j}t\right)  ^{-1} & \text{for }j=2
\end{array}
\right.  . \label{gammaj}%
\end{equation}
For $j=2$, the contour does not encircle $1/\eta_{2},$ and the representation
is valid for $\eta_{2}$ such that $\left\vert \arg\left(  1-\eta_{2}\right)
\right\vert <\pi$. We obtain the following lemma by deriving a saddle-point
approximation to the integral in (\ref{integralrepF11}). The relevant saddle
points are%
\begin{equation}
t_{j}=\left\{
\begin{array}
[c]{cl}%
\frac{1}{2\eta_{j}}\left\{  \eta_{j}-1+\sqrt{\left(  \eta_{j}-1\right)
^{2}+4\varepsilon\eta_{j}}\right\}  & \text{for }j=1\\
\frac{1}{2\eta_{j}\left(  \varepsilon-1\right)  }\left\{  -1+\sqrt
{1+4\varepsilon\left(  \varepsilon-1\right)  \eta_{j}}\right\}  & \text{for
}j=2
\end{array}
\right.  . \label{tj}%
\end{equation}


We shall need the following additional notation. Let
\begin{equation}
\omega_{j}=\arg\varphi_{j}^{\prime\prime}\left(  t_{j}\right)  +\pi\text{ and
}\omega_{0j}=\arg\left(  t_{j}-1\right)  , \label{omegaj}%
\end{equation}
where the branches of $\arg\left(  \cdot\right)  $ are chosen so that
$\left\vert \omega_{j}+2\omega_{0j}\right\vert \leq\pi/2.$ Further, for any
small $\delta>0$ let $\Omega_{1\delta}$ be the set of $\left(  \varepsilon
,\eta_{1}\right)  \in\mathbb{R}\times\mathbb{C}$ such that $\delta
\leq\varepsilon-1\leq1/\delta,$ and%
\[
\operatorname{Re}\eta_{1}\geq-2\varepsilon+1,\text{ }\mathrm{dist}\left(
\eta_{1},\mathbb{R}\backslash\left[  0,\infty\right)  \right)  \geq
\delta,\text{ }\left\vert \eta_{1}\right\vert \leq1/\delta.
\]
Similarly, let $\Omega_{2\delta}$ be the set of $\left(  \varepsilon,\eta
_{2}\right)  \in\mathbb{R}\times\mathbb{C}$ such that $\delta\leq
\varepsilon-1\leq1/\delta,$ and%
\[
\mathrm{dist}\left(  \eta_{2},\mathbb{R}\backslash\left[  0,1\right]  \right)
\geq\delta,\text{ }\left\vert \eta_{2}\right\vert \leq1/\delta.
\]
Here, for any $A\subseteq\mathbb{C}$ and $B\subseteq\mathbb{C}$,
$\mathrm{dist}\left(  A,B\right)  =\inf_{a\in A,b\in B}\left\vert
a-b\right\vert .$ Figure \ref{uniform} shows cross-sections of $\Omega
_{1\delta}$ and $\Omega_{2\delta}$ for fixed $\varepsilon.$%

%TCIMACRO{\FRAME{ftbpFU}{5.6896in}{2.6282in}{0pt}{\Qcb{Cross-sections of the
%sets $\Omega_{j\delta}$ for $\varepsilon=2$ and $\delta=0.1.$ The horizontal
%and vertical axes correspond to the real and purely imaginary numbers,
%respectively.}}{\Qlb{uniform}}{uniform.eps}%
%{\special{ language "Scientific Word";  type "GRAPHIC";
%maintain-aspect-ratio TRUE;  display "USEDEF";  valid_file "F";
%width 5.6896in;  height 2.6282in;  depth 0pt;  original-width 9.4714in;
%original-height 4.3613in;  cropleft "0";  croptop "1";  cropright "1";
%cropbottom "0";  filename 'uniform.eps';file-properties "XNPEU";}} }%
%BeginExpansion
\begin{figure}[ptb]%
\centering
\includegraphics[
height=2.6282in,
width=5.6896in
]%
{uniform.eps}%
\caption{Cross-sections of the sets $\Omega_{j\delta}$ for $\varepsilon=2$ and
$\delta=0.1.$ The horizontal and vertical axes correspond to the real and
purely imaginary numbers, respectively.}%
\label{uniform}%
\end{figure}
%EndExpansion


\begin{lemma}
\label{1F1approximation}As $m\rightarrow\infty,$ we have for $j=1,2$%
\begin{equation}
F_{j}=C_{m}\psi_{j}\left(  t_{j}\right)  e^{-\mathrm{i}\omega_{j}/2}\left\vert
2\pi m\varphi_{j}^{\prime\prime}\left(  t_{j}\right)  \right\vert ^{-1/2}%
\exp\left\{  -m\varphi_{j}\left(  t_{j}\right)  \right\}  \left(
1+o(1)\right)  . \label{ConfluentApprox}%
\end{equation}
The convergence $o(1)\rightarrow0$ holds uniformly with respect to $\left(
\varepsilon,\eta\right)  \in\Omega_{j\delta}$ for any $\delta>0$.
\end{lemma}

Point-wise asymptotic approximation (\ref{ConfluentApprox}) was established in
Passemier et al (2014) for $j=1,$ and in Paris (2013a,b) for $j=2.$ However,
those papers do not study the uniformity of the approximation error, which is
important for our analysis. A proof of Lemma \ref{1F1approximation} is
available from the authors upon request. We shall report it elsewhere.

Using Lemmas \ref{uniform1}, \ref{1F1approximation}, and Stirling's
approximation%
\begin{equation}
C_{m}=\frac{\sqrt{\pi p\left(  1-c_{1}\right)  }}{r}\exp\left\{  m\left(
\varepsilon-1\right)  \ln\left(  \varepsilon-1\right)  -m\varepsilon
\ln\varepsilon\right\}  \left(  1+o(1)\right)  \label{StirlingCm}%
\end{equation}
we set the components of the \textquotedblleft Laplace form\textquotedblright%
\ (\ref{LaplaceForm}) of $\left.  _{\bar{p}}F_{\bar{q}}\right.  $ for cases
REG$_{0},$ REG, and CCA as follows%
\begin{equation}
2f_{III}(z)=\left\{
\begin{array}
[c]{ll}%
\frac{1-c_{1}}{c_{1}}\varphi_{0}\left(  t_{0}\right)  & \text{for REG}_{0}\\
\frac{1-c_{1}}{c_{1}}\left(  \varphi_{j}\left(  t_{j}\right)  +\varepsilon
\ln\varepsilon-\left(  \varepsilon-1\right)  \ln\left(  \varepsilon-1\right)
\right)  \text{ } & \text{for REG and CCA}%
\end{array}
\right.  \label{f2}%
\end{equation}
and%
\begin{equation}
g_{III}(z)=\left\{
\begin{array}
[c]{ll}%
\left(  1+4\eta_{0}\right)  ^{-1/4}\left(  1+o\left(  1\right)  \right)  &
\text{for REG}_{0}\\
\sqrt{c_{1}/r^{2}}e^{-\mathrm{i}\omega_{j}/2}\left\vert \varphi_{j}%
^{\prime\prime}\left(  t_{j}\right)  \right\vert ^{-1/2}\psi_{j}\left(
t_{j}\right)  \left(  1+o\left(  1\right)  \right)  & \text{for REG and CCA}%
\end{array}
\right.  . \label{g2}%
\end{equation}
To express $t_{j}$ and $\eta_{j}$ in terms of $z,$ one should use (\ref{tj})
and (\ref{etas}). We do not need to know how exactly $o\left(  1\right)  $ in
(\ref{g2}) depend on $z$. For our purposes, the knowledge of the fact that
$o\left(  1\right)  $ are analytic functions of $\eta_{j}$ that converge to
zero uniformly with respect to $\left(  \varepsilon,\eta_{j}\right)  \in
\Omega_{j\delta}$ is sufficient. The analyticity of $o(1)$ follows from the
analyticity of the functions on the left hand sides, and of the factors of
$1+o(1)$ on the right hand sides of the equations (\ref{BesselApprox}) and
(\ref{ConfluentApprox}).

Using the definitions of $\varphi_{j}$ and $t_{j},$ it is straightforward to
verify that $f_{III}^{(REG)}(z)$ and $f_{III}^{(CCA)}(z)$ converge to
$f_{III}^{(REG_{0})}(z)$ as $c_{2}\rightarrow0$. Since, as has been shown
above, a similar convergence holds for $f_{I}$ and $f_{II},$ we have
$f^{(REG)}(z),f^{(CCA)}(z)\rightarrow f^{(REG_{0})}(z)$ as $c_{2}\rightarrow
0$. Elementary calculations that use equations (\ref{f1Case}), (\ref{phi0}),
(\ref{f2}) together with the explicit forms of $f_{I}^{(REG_{0})}$ and
$f_{I}^{(SMD)}$ given in Table \ref{Table 2a} show that $f^{(REG_{0}%
)}(z)\rightarrow f^{(SMD)}(z)$ as $c_{1}\rightarrow0$ after transformations
$\theta\mapsto\sqrt{c_{1}}\theta$ and $z\mapsto\sqrt{c_{1}}z+1$.

\subsection{Contours of steep descent}

We shall now show how to deform contours $\mathcal{K}$ in (\ref{Laplace form})
into the contours of steep descent. First, we find saddle points of functions
$f(z)$ for each of the six cases. Note that the derivative of $f_{II}(z)$
equals minus half of the Stieltjes transform $m_{\mathbf{c}}\left(  z\right)
$ of the corresponding limiting spectral distribution $F_{\mathbf{c}}$.
Although the Stieltjes transform is formally defined on $\mathbb{C}^{+}$, the
definition remains valid on the part of the real line outside the support
$\left[  b_{-},b_{+}\right]  $ of $F_{\mathbf{c}}$. Since we assume that
$\gamma_{1}\leq1,$ $F_{\mathbf{c}}$ does not have any non-trivial mass at $0$
for sufficiently large $\mathbf{n}$ and~$p$.

To find saddle points of $f(z)$ we solve equation%
\begin{equation}
m_{\mathbf{c}}\left(  z\right)  =2\mathrm{d}f_{III}(z)/\mathrm{d}z.
\label{extremum points}%
\end{equation}
In the Appendix, we find real solutions to (\ref{extremum points}), $z_{0}$,
that satisfy inequality $z_{0}>b_{+}$. These solutions are reported in the
following lemma.

\begin{lemma}
\label{critical}Let $b_{+}$ be the upper boundary of support of $F_{\mathbf{c}%
},$ and $\bar{\theta}$ be the threshold corresponding to $F_{\mathbf{\gamma}}$
as given in Table \ref{Table 3}. Then, for $\theta\in\left(  0,\bar{\theta
}\right)  $ and sufficiently large $\mathbf{n}$ and $p$ as $\mathbf{n}%
,p\rightarrow_{\mathbf{\gamma}}\infty,$%
\begin{equation}
z_{0}=\left\{
\begin{array}
[c]{ll}%
\theta+1/\theta & \text{for SMD}\\
\left(  1+\theta\right)  \left(  \theta+c_{1}\right)  /\theta & \text{for PCA
and REG}_{0}\\
\left(  1+\theta\right)  \left(  \theta+c_{1}\right)  /\left[  \theta l\left(
\theta\right)  \right]  & \text{for SigD, REG, and CCA}%
\end{array}
\right.  \label{zcase}%
\end{equation}
satisfy inequality $z_{0}>b_{+}$ and solve equation (\ref{extremum points}).
\end{lemma}

As $c_{2}\rightarrow0$ while $c_{1}$ stays constant, the value of $z_{0}$ for
SigD, REG, and CCA converges to that for PCA and REG$_{0}.$ The latter value
in its turn converges to the value of $z_{0}$ for SMD when $c_{1}\rightarrow
0$, after the transformations $\theta\mapsto\sqrt{c_{1}}\theta$ and
$z_{0}\mapsto\sqrt{c_{1}}z_{0}+1$. Precisely, solving equation%
\[
\sqrt{c_{1}}z_{0}+1=\left(  1+\sqrt{c_{1}}\theta\right)  \left(  \sqrt{c_{1}%
}\theta+c_{1}\right)  /\left(  \sqrt{c_{1}}\theta\right)
\]
for $z_{0}$ and taking limit as $c_{1}\rightarrow0$ yields $z_{0}%
=\theta+1/\theta$.

For the rest of the paper, assume that $\theta\in\left(  0,\bar{\theta
}\right)  $. We deform contour $\mathcal{K}$ in (\ref{Laplace form}) so that
it passes through the saddle point $z_{0}$ as follows. Let $\mathcal{K}%
=\mathcal{K}_{+}\cup\mathcal{K}_{-},$ where $\mathcal{K}_{-}$ is the complex
conjugate of $\mathcal{K}_{+}$ and $\mathcal{K}_{+}=\mathcal{K}_{1}%
\cup\mathcal{K}_{2}.$ For SMD, PCA, and SigD, let%
\begin{align}
\mathcal{K}_{1}  &  =\left\{  z_{0}+\mathrm{i}t:0\leq t\leq2z_{0}\right\}
\text{ and}\label{contours}\\
\mathcal{K}_{2}  &  =\left\{  x+\mathrm{i}2z_{0}:-\infty<x\leq z_{0}\right\}
. \label{contours 1}%
\end{align}
The deformed contour is shown on Figure \ref{Case 0 contour}.
%TCIMACRO{\FRAME{ftbpFU}{3.8908in}{3.384in}{0pt}{\Qcb{Deformed contour
%$\QTR{cal}{K}$ for SMD, PCA, and SigD.}}{\Qlb{Case 0 contour}}%
%{contour_distortion.eps}{\special{ language "Scientific Word";
%type "GRAPHIC";  maintain-aspect-ratio TRUE;  display "USEDEF";
%valid_file "F";  width 3.8908in;  height 3.384in;  depth 0pt;
%original-width 9.6522in;  original-height 8.3861in;  cropleft "0";
%croptop "1";  cropright "1";  cropbottom "0";
%filename 'contour_distortion.eps';file-properties "XNPEU";}} }%
%BeginExpansion
\begin{figure}[ptb]%
\centering
\includegraphics[
height=3.384in,
width=3.8908in
]%
{contour_distortion.eps}%
\caption{Deformed contour $\mathcal{K}$ for SMD, PCA, and SigD.}%
\label{Case 0 contour}%
\end{figure}
%EndExpansion


Note that the singularities of the integrand in (\ref{Laplace form}) are
situated at $z=\lambda_{j}$ (plus an additional singularity at $z=c_{1}%
(1+\theta)/\left(  \theta c_{2}\right)  <z_{0}$ for SigD). Since $\lambda
_{1}\overset{a.s.}{\rightarrow}\beta_{+}$ and $z_{0}>b_{+},$ inequality
$z_{0}>\lambda_{1}$ must hold with probability approaching one as
$\mathbf{n},p\rightarrow_{\mathbf{\gamma}}\infty.$ Therefore by Cauchy's
theorem, the deformation of the contour does not change the value of $L\left(
\theta;\Lambda\right)  $ with probability approaching one as $\mathbf{n}%
,p\rightarrow_{\mathbf{\gamma}}\infty.$

Strictly speaking, the deformation of the contour is not continuous because
$\mathcal{K}_{+}$ does not approach\ $\mathcal{K}_{-}$ at $-\infty$. In
particular, in contrast to the original contour, the deformed one is not
\textquotedblleft closed\textquotedblright\ at $-\infty$. Nevertheless, such
an \textquotedblleft opening up\textquotedblright\ at $-\infty$ does not lead
to the change of the value of the integral because the integrand converges
fast to zero by absolute value as $\operatorname{Re}z\rightarrow-\infty$.

\begin{remark}
In the event of asymptotically negligeable probability that the deformed
contour $\mathcal{K}$ does not encircle all $\lambda_{j},$ we not only loose
the equality (\ref{Laplace form}) but also face the difficulty that function
$g(z)$ ceases to be well defined as the definition of $g_{II}(z)$ contains a
logarithm of a non-positive number. To eliminate any ambiguity, if such an
event holds we shall redefine $g_{II}(z)$ as unity.
\end{remark}

For REG$_{0}$ and CCA, let%
\[
z_{1}=\left\{
\begin{array}
[c]{ll}%
-\left(  1-c_{1}\right)  ^{2}/\left[  4\theta\right]  & \text{for REG}_{0}\\
-c_{1}\left(  1-c_{1}\right)  ^{2}l\left(  \theta\right)  /\left[  4\theta
r^{2}\right]  & \text{for CCA}%
\end{array}
\right.  ,
\]
and let%
\begin{align*}
\mathcal{K}_{1}  &  =\left\{  z_{1}+\left\vert z_{0}-z_{1}\right\vert
\exp\left\{  \mathrm{i}\gamma\right\}  :\gamma\in\left[  0,\pi/2\right]
\right\}  \text{ and}\\
\mathcal{K}_{2}  &  =\left\{  z_{1}-x+\left\vert z_{0}-z_{1}\right\vert
\exp\left\{  \mathrm{i}\pi/2\right\}  :x\geq0\right\}  .
\end{align*}
The corresponding contour $\mathcal{K}$ is shown on Figure
\ref{contourgeneral}. Similarly to the SMD, PCA and SigD cases, the
deformation of the contour in (\ref{Laplace form}) to $\mathcal{K}$ does not
change the value of $L\left(  \theta;\Lambda\right)  $ with probability
approaching one as $\mathbf{n},p\rightarrow_{\mathbf{\gamma}}\infty.$%

%TCIMACRO{\FRAME{ftbpFU}{3.8856in}{3.5812in}{0pt}{\Qcb{Deformed contour
%$\QTR{cal}{K}$ for REG$_{0}$ and CCA.}}{\Qlb{contourgeneral}}%
%{contourgeneral.eps}{\special{ language "Scientific Word";  type "GRAPHIC";
%maintain-aspect-ratio TRUE;  display "USEDEF";  valid_file "F";
%width 3.8856in;  height 3.5812in;  depth 0pt;  original-width 8.9301in;
%original-height 8.2313in;  cropleft "0";  croptop "1";  cropright "1";
%cropbottom "0";  filename 'contourgeneral.eps';file-properties "XNPEU";}} }%
%BeginExpansion
\begin{figure}[ptb]%
\centering
\includegraphics[
height=3.5812in,
width=3.8856in
]%
{contourgeneral.eps}%
\caption{Deformed contour $\mathcal{K}$ for REG$_{0}$ and CCA.}%
\label{contourgeneral}%
\end{figure}
%EndExpansion


For REG, deformed contour $\mathcal{K}$ in $z$-plane is simpler to describe as
an image of a contour $\mathcal{C}$ in $\tau$-plane, where $\tau=\eta_{1}%
t_{1}$ with%
\begin{equation}
\eta_{1}=z\theta c_{2}/\left[  c_{1}\left(  1-c_{1}\right)  \right]
\label{eta1}%
\end{equation}
and $t_{1}$ as defined in (\ref{tj}). Let $\mathcal{C}=\mathcal{C}_{+}%
\cup\mathcal{C}_{-},$ where $\mathcal{C}_{-}$ is the complex conjugate of
$\mathcal{C}_{+}$ and $\mathcal{C}_{+}=\mathcal{C}_{1}\cup\mathcal{C}_{2},$
and let%
\begin{align*}
\mathcal{C}_{1}  &  =\left\{  -\varepsilon+\left\vert \tau_{0}+\varepsilon
\right\vert \exp\left\{  \mathrm{i}\gamma\right\}  :\gamma\in\left[
0,\pi/2\right]  \right\}  \text{ and}\\
\mathcal{C}_{2}  &  =\left\{  -\varepsilon-x+\left\vert \tau_{0}%
+\varepsilon\right\vert \exp\left\{  \mathrm{i}\pi/2\right\}  :x\geq0\right\}
,
\end{align*}
where $\tau_{0}=\left(  \theta+c_{1}\right)  /\left(  1-c_{1}\right)  .$

Using (\ref{eta1}) and the identity
\begin{equation}
\eta_{1}=\tau\left(  \tau+1\right)  /(\tau+\varepsilon),
\label{identity for eta1}%
\end{equation}
we obtain%
\begin{equation}
z=\frac{c_{1}\left(  1-c_{1}\right)  }{\theta c_{2}}\frac{\tau\left(
\tau+1\right)  }{\tau+\varepsilon}. \label{transformation}%
\end{equation}
We define the deformed contour $\mathcal{K}$ in $z$-plane as the image of
$\mathcal{C}$ under the trasformation $\tau\rightarrow z$ given by
(\ref{transformation}). The parts $\mathcal{K}_{+},\mathcal{K}_{-}%
,\mathcal{K}_{1}$ and $\mathcal{K}_{2}$ of $\mathcal{K}$ are defined as the
images of the corresponding parts of $\mathcal{C}$. Note that $\tau_{0}$ is
transformed to $z_{0}$ so that $\mathcal{K}$ passes through the saddle point
$z_{0}.$

The following lemma is proven in the Appendix. It shows that $\mathcal{K}_{1}$
are contours of steep descent of $-\operatorname{Re}f\left(  z\right)  $ for
all the six cases, SMD, PCA, SigD, REG$_{0}$, REG, and CCA.

\begin{lemma}
\label{ContourProp1F1} For any of the six cases that we study, as $z$ moves
along the corresponding $\mathcal{K}_{1}$ away from $z_{0}$,
$-\operatorname{Re}f\left(  z\right)  $ is strictly decreasing.
\end{lemma}

\section{Laplace approximation}

The goal of this section is to derive Laplace approximations to the integrals%
\[
L\left(  \theta;\Lambda\right)  =\sqrt{\pi p}\frac{1}{2\pi\mathrm{i}}%
\int_{\mathcal{K}}\exp\left\{  -pf(z)\right\}  g(z)\mathrm{d}z
\]
for the six cases that we study. First, consider a general integral%
\[
I_{p,\omega}=\int_{\mathcal{K}_{p,\omega}}e^{-p\phi_{p,\omega}(z)}%
\chi_{p,\omega}(z)\mathrm{d}z,
\]
where $p\rightarrow\infty,$ $\omega$ is a $k$-dimensional parameter that
belongs to a subset $\Omega$ of $\mathbb{R}^{k},$ $\mathcal{K}_{p,\omega}$ is
a path in $\mathbb{C}$ that starts at $a_{p,\omega}$ and ends at $b_{p,\omega
}$, and for sufficiently large $p,$ $\phi_{p,\omega}(z)$ is a single-valued
holomorphic function of $z$ in a domain $T_{p,\omega}$ that contains
$\mathcal{K}_{p,\omega}$.

We allow $\chi_{p,\omega}(z)$ to be a random element of the normed space of
continuous functions on $\mathcal{K}_{p,\omega}$ with the supremum norm.
Furthermore, we suppose that for any $\delta>0$, there exists $\bar{p}$ such
that for any $p>\bar{p},$ $\chi_{p,\omega}(z)$ is a single-valued holomorphic
function of $z$ in the domain $T_{p,\omega}$ with probability larger than
$1-\delta$. In what follows, we shall omit subscripts $p$ and $\omega$ from
the notation $\phi_{p,\omega},$ $\chi_{p,\omega},$ $\mathcal{K}_{p,\omega},$
etc. to make it lighter.

Suppose that $\phi^{\prime}\left(  z\right)  =0$ at $z_{0}$ which is an
interior point of $\mathcal{K},$ and suppose that $\operatorname{Re}\phi(z)$
is strictly increasing as $z$ moves away from $z_{0}$ along the path. In other
words, the path $\mathcal{K}$ is a contour of steep descent\ of
$-\operatorname{Re}\phi(z)$. Denote a closed segment of $\mathcal{K}$
contained between $z_{1}$ and $z_{2}$ as $\left[  z_{1},z_{2}\right]
_{\mathcal{K}}$. Similarly denote the segments that exclude one or both
endpoints as $\left[  z_{1},z_{2}\right)  _{\mathcal{K}},$ $\left(
z_{1},z_{2}\right]  _{\mathcal{K}},$ and $\left(  z_{1},z_{2}\right)
_{\mathcal{K}}$. Let $\beta$ be the limiting value of $\arg\left(
z-z_{0}\right)  $ on the principal branch as $z\rightarrow z_{0}$ along
$\left(  z_{0},b\right)  _{\mathcal{K}}$. Finally, let $\phi_{s}$ and
$\chi_{s}$ with $s=0,1,...$ be the coefficients in the power series
representations%
\begin{equation}
\phi\left(  z\right)  =\sum_{s=0}^{\infty}\phi_{s}\left(  z-z_{0}\right)
^{s},\text{ }\chi(z)=\sum_{s=0}^{\infty}\chi_{s}\left(  z-z_{0}\right)  ^{s}.
\label{f and g series}%
\end{equation}


We assume that there exist positive constants $C_{1},...,C_{4}$ that do not
depend on $p$ and on $\omega,$ such that for all $\omega\in\Omega$, for
sufficiently large $p:$

\begin{enumerate}
\item[A0] The length of the path $\mathcal{K}$ is bounded, uniformly over
$\omega\in\Omega$ and all sufficiently large $p.$ Furthermore,%
\[
\sup_{z\in\left(  z_{0},b\right)  _{\mathcal{K}}}\left\vert z-z_{0}\right\vert
>C_{1},\text{ and }\sup_{z\in\left(  a,z_{0}\right)  _{\mathcal{K}}}\left\vert
z-z_{0}\right\vert >C_{1}%
\]


\item[A1] Functions $\phi\left(  z\right)  $ and $\chi(z)$ are holomorphic in
the ball $\left\vert z-z_{0}\right\vert \leq C_{1}$

\item[A2] The coefficient $\phi_{2}$ satisfies $C_{2}\leq\left\vert \phi
_{2}\right\vert \leq C_{3}$

\item[A3] The third derivative of $\phi\left(  z\right)  $ satisfies
inequality
\[
\sup_{\left\vert z-z_{0}\right\vert \leq C_{1}}\left\vert \mathrm{d}^{3}%
\phi\left(  z\right)  /\mathrm{d}z^{3}\right\vert \leq C_{4}%
\]


\item[A4] For any positive $\varepsilon<C_{1},$ which does not depend on $p$
and $\omega$, and for all $z_{1}\in\mathcal{K}$ such that $\left\vert
z_{1}-z_{0}\right\vert =\varepsilon,$ there exist positive constants
$C_{5},C_{6}$, such that%
\[
\operatorname{Re}\left(  \phi\left(  z_{1}\right)  -\phi_{0}\right)
>C_{5}\text{ and }\left\vert \operatorname{Im}\left(  \phi\left(
z_{1}\right)  -\phi_{0}\right)  \right\vert <C_{6}%
\]


\item[A5] For a subset $\Theta$ of $\mathbb{C}$ that consists of all points
whose Euclidean distance from $\mathcal{K}$ is no larger than $C_{1},$%
\[
\sup_{z\in\Theta}\left\vert \chi(z)\right\vert =O_{\mathrm{P}}(1)
\]
as $p\rightarrow\infty,$ where $O_{\mathrm{P}}(1)$ is uniform in $\omega
\in\Omega$.
\end{enumerate}

The following lemma is a fairly straightforward extension of Theorem 7.1 of
Olver (1997), p. 127 to the situation where functions $\phi(z),$ $\chi(z)$ and
the contour $\mathcal{K}$ depend on $p$ and $\omega.$ In Olver's original
theorem, which uses different notation, both the functions and the contour are
fixed. A proof of the extension is available from the authors upon request.

\begin{lemma}
\label{Olver}Under assumptions A0-A5, for \textit{any positive integer
}$\kappa$, as $p\rightarrow\infty,$ we have%
\[
I_{p,\omega}=2e^{-p\phi_{0}}\left[  \sum_{s=0}^{\kappa-1}\Gamma\left(
s+\frac{1}{2}\right)  \frac{a_{2s}}{p^{s+1/2}}+\frac{O_{\mathrm{P}}\left(
1\right)  }{p^{\kappa+1/2}}\right]  ,
\]
\textit{where }$O_{\mathrm{P}}\left(  1\right)  $\textit{\ is uniform in
}$\omega\in\Omega$ and \textit{the coefficients }$a_{j}$\textit{\ can be
expressed through }$\phi_{i}$ \textit{and }$\chi_{i}$ \textit{defined above.
In particular we have }$a_{0}=\chi_{0}/[2\phi_{2}^{1/2}],$ \textit{where
}$\phi_{2}^{1/2}=\exp\left\{  \left(  \log\left\vert \phi_{2}\right\vert
+\arg\phi_{2}\right)  /2\right\}  $ with the branch of $\arg\phi_{2}$ chosen
so that $\left\vert \arg\phi_{2}+2\beta\right\vert \leq\pi/2.$
\end{lemma}

We use Lemma \ref{Olver} to obtain the Laplace approximation to
\begin{equation}
L_{1}\left(  \theta;\Lambda\right)  =\sqrt{\pi p}\frac{1}{2\pi\mathrm{i}}%
\int_{\mathcal{K}_{1}\cup\mathcal{\bar{K}}_{1}}e^{-pf(z)}g(z)\mathrm{d}z.
\label{I1case}%
\end{equation}
Then we show that $L_{1}\left(  \theta;\Lambda\right)  $ asymptotically
dominates the \textquotedblleft residual\textquotedblright\ $L\left(
\theta;\Lambda\right)  -L_{1}\left(  \theta;\Lambda\right)  $. For this
analysis, it is important to know the values of $f(z_{0})$ and $\mathrm{d}%
^{2}f(z_{0})/\mathrm{d}z^{2}$. We derive them in the Appendix. It turns out
that as long as $\theta\in\left[  0,\bar{\theta}\right)  $, $f(z_{0})=0$ for
all the six cases that we study. The values of $\mathrm{d}^{2}f(z_{0}%
)/\mathrm{d}z^{2}$ are all negative. The explicit form of $D_{2}\equiv
\theta^{2}\left(  -2\mathrm{d}^{2}f(z_{0})/\mathrm{d}z^{2}\right)  ^{-1},$
which is somewhat shorter than that for $\mathrm{d}^{2}f(z_{0})/\mathrm{d}%
z^{2}$ is reported in Table \ref{Table 5}. We formulate the main result of
this section in the following theorem. Its proof is given in the Appendix.%

%TCIMACRO{\TeXButton{B}{\begin{table}[tbp] \centering}}%
%BeginExpansion
\begin{table}[tbp] \centering
%EndExpansion%
\begin{tabular}
[c]{llll}%
Case & Value of $D_{2}$\bigskip & Case & Value of $D_{2}$\\\hline
SMD & $1-\theta^{2}\bigskip$ & REG$_{0}$ & $c_{1}\left(  1+c_{1}%
+2\theta\right)  \left(  c_{1}-\theta^{2}\right)  \bigskip$\\
PCA & $c_{1}\left(  c_{1}-\theta^{2}\right)  \left(  1+\theta\right)
^{2}\bigskip$ & REG & $c_{1}h\left(  c_{1}+\theta+\left(  1+\theta\right)
l\right)  /l^{4}\bigskip$\\
SigD & $r^{2}h\left(  1+\theta\right)  ^{2}/l^{4}$ & CCA & $c_{1}^{2}h\left(
2\left(  c_{1}+\theta\right)  +l\left(  1-c_{1}\right)  \right)  /\left(
l^{3}\left(  c_{1}+c_{2}\right)  \right)  $\\\hline
\end{tabular}
\caption{The values of $D_2\equiv\theta^{2}(-2\mathrm{d}^{2}f(z_0)/\mathrm{d}z^{2})^{-1}$. Here $l \equiv l(\theta)$ is
as defined in (\ref{loftheta}) and $h \equiv h(\theta)=c_1+c_2(1+\theta)^2-\theta^2$.}\label{Table 5}%
%TCIMACRO{\TeXButton{E}{\end{table}}}%
%BeginExpansion
\end{table}%
%EndExpansion


\begin{theorem}
\label{Icase}Suppose that the null hypothesis holds, that is, $\theta_{0}=0$.
Let $\bar{\theta}$ be the threshold corresponding to $F_{\mathbf{\gamma}}$ as
given in Table \ref{Table 3}, and let $\varepsilon$ be an arbitrarily small
fixed positive number. Then for \textit{any }$\theta\in\left(  0,\bar{\theta
}-\varepsilon\right]  ,$\textit{\ as }$\mathbf{n},p\rightarrow_{\mathbf{\gamma
}}\infty$,\textit{\ we have}%
\begin{equation}
L\left(  \theta;\Lambda\right)  =\frac{g(z_{0})}{\sqrt{-2\mathrm{d}^{2}%
f(z_{0})/\mathrm{d}z^{2}}}+O_{\mathrm{P}}\left(  p^{-1}\right)  ,
\label{Icase equality}%
\end{equation}
\textit{where }$O_{\mathrm{P}}\left(  p^{-1}\right)  $\textit{\ is uniform in
}$\theta\in\left(  0,\bar{\theta}-\varepsilon\right]  $ and the principal
branch of the square root is taken.
\end{theorem}

\section{Asymptotics of LR}

Combining the results of Theorem \ref{Icase} with the definitions of $g(z)$
and the values of $-2\mathrm{d}^{2}f(z_{0})/\mathrm{d}z^{2}$ (given in Table
\ref{Table 5}), it is straightforward to establish the following theorem. Let
\[
\Delta_{p}(\theta)=p\int\ln\left(  z_{0}-\lambda\right)  \mathrm{d}\left(
\hat{F}\left(  \lambda\right)  -F_{\mathbf{c}}\left(  \lambda\right)  \right)
.
\]
In accordance with the remark made above, we define $\Delta_{p}(\theta)$ as
zero in the event of asymptotically negligeable probability that $z_{0}%
\leq\lambda_{1}$.

\begin{theorem}
\label{LR asymptotics}Suppose that the null hypothesis holds, that is
$\theta_{0}=0$. Let $\bar{\theta}$ be the threshold corresponding to
$F_{\mathbf{\gamma}}$ as given in Table \ref{Table 3}, and let $\varepsilon$
be an arbitrarily small fixed positive number. Then for \textit{any }%
$\theta\in\left(  0,\bar{\theta}-\varepsilon\right]  ,$\textit{\ as
}$\mathbf{n},p\rightarrow_{\mathbf{\gamma}}\infty$,\textit{\ we have}%
\[
L\left(  \theta;\Lambda\right)  =\exp\left\{  -\frac{1}{2}\Delta_{p}%
(\theta)+\frac{1}{2}\ln\left(  1-\left[  \delta_{p}\left(  \theta\right)
\right]  ^{2}\right)  \right\}  \left(  1+o_{\mathrm{P}}(1)\right)  ,
\]
where
\[
\delta_{p}\left(  \theta\right)  =\left\{
\begin{array}
[c]{ll}%
\theta & \text{for SMD}\\
\theta/\sqrt{c_{1}} & \text{for PCA and REG}_{0}\\
\theta r/\left(  c_{1}l\left(  \theta\right)  \right)  & \text{for SigD, REG,
and CCA}%
\end{array}
\right.
\]
and $o_{\mathrm{P}}(1)$ is \textit{uniform in }$\theta\in\left(  0,\bar
{\theta}-\varepsilon\right]  $.
\end{theorem}

Statistic $\Delta_{p}(\theta)$ is a linear spectral statistic. As follows from
the CLT derived by Bai and Yao (2005), Bai and Silverstein (2004), and Zheng
(2012) for the semi-circle, Marchenko-Pastur, and Wachter limiting
distributions $F_{\mathbf{c}}\,,$ respectively, statistic $\Delta_{p}(\theta)$
weakly converges to a Gaussian process indexed by $\theta\in\left(
0,\bar{\theta}-\varepsilon\right]  .$ The explicit form of the mean and the
covariance structure can be obtained from the general formulae for the
asymptotic mean and covariance of linear spectral statistics given in Theorem
1.1 of Bai and Yao (2005) for SMD, in Theorem 1.1 of Bai and Silverstein
(2004) for PCA and REG$_{0}$, and in Theorem 4.1 and Example 4.1 of Zheng
(2012) for the remaining cases. For PCA, the corresponding calculations have
been done in Onatski et al (2013). We omit details of the similar calculations
for the remaining cases to save space. The convergence of $\Delta_{p}(\theta)$
and Theorem \ref{LR asymptotics} imply the following theorem.

\begin{theorem}
\label{AsymptoticNormality}Suppose that the null hypothesis holds, that is
$\theta_{0}=0$. Let $\bar{\theta}$ be the threshold corresponding to
$F_{\mathbf{\gamma}}$ as given in Table \ref{Table 3}, and let $\varepsilon$
be an arbitrarily small fixed positive number. Further, let $C\left[
0,\bar{\theta}-\varepsilon\right]  $ be the space of continuous functions on
$\left[  0,\bar{\theta}-\varepsilon\right]  $ equipped with the supremum norm.
Then $\ln L\left(  \theta;\Lambda\right)  $ viewed as random elements of
$C\left[  0,\bar{\theta}-\varepsilon\right]  $ converge weakly to
$\mathcal{L}\left(  \theta\right)  $ with Gaussian finite dimensional
distributions such that%
\[
\mathbb{E}\mathcal{L}\left(  \theta\right)  =\frac{1}{4}\ln\left(  1-\left[
\delta\left(  \theta\right)  \right]  ^{2}\right)
\]
and%
\[
\mathbb{C}ov\left(  \mathcal{L}\left(  \theta_{1}\right)  ,\mathcal{L}\left(
\theta_{2}\right)  \right)  =-\frac{1}{2}\ln\left(  1-\delta\left(  \theta
_{1}\right)  \delta\left(  \theta_{2}\right)  \right)
\]
with%
\[
\delta\left(  \theta\right)  =\left\{
\begin{array}
[c]{ll}%
\theta & \text{for SMD}\\
\theta/\sqrt{\gamma_{1}} & \text{for PCA and REG}_{0}\\
\theta\rho/\left(  \gamma_{1}+\gamma_{2}+\theta\gamma_{2}\right)  & \text{for
SigD, REG, and CCA}%
\end{array}
\right.  .
\]
Here $\rho,\gamma_{1},\gamma_{2}$ are the limits of $r,c_{1},c_{2}$ as
$\mathbf{n},p\rightarrow_{\mathbf{\gamma}}\infty$.
\end{theorem}

Note that the theorem establishes the weak convergence of the log likelihood
ratio viewed as a random element of the space of continuous functions. This is
much stronger than simply the convergence of the finite dimensional
distributions of the log likelihood process. In particular, the theorem can be
used to find the asymptotic distribution of the supremum of the likelihood
ratio, and thus, to find the asymptotic critical values of the likelihood
ratio test. We do not pursue this line of research here.

Let $\left\{  \mathbb{P}_{p,\theta}\right\}  $ and $\left\{  \mathbb{P}%
_{p,0}\right\}  $ be the sequences of measures corresponding to the joint
distributions of $\lambda_{1},...,\lambda_{p}$ when $\theta_{0}=\theta$ and
when $\theta_{0}=0$ respectively. Then Theorem \ref{AsymptoticNormality}
implies, via Le Cam's first lemma, the mutual contiguity of $\left\{
\mathbb{P}_{p,\theta}\right\}  $ and $\left\{  \mathbb{P}_{p,0}\right\}  $ as
$\mathbf{n},p\rightarrow_{\mathbf{\gamma}}\infty$. This reveals the
statistical meaning of the phase transition thresholds as the upper boundaries
of the contiguity regions for spiked models.

The precise form of the autocovariance of $\mathcal{L}\left(  \theta\right)  $
shows that,\footnote{Fyodorov et al (2013) have an interesting discussion of
ubiquity of random processes with logarithmic covariance structure in physics
and engineering applications. In their paper, such processes appear as
limiting objects related to the behavior of the characteristic polynomials of
large matrices from Gaussian Unitary Ensemble.} although the experiment of
observing $\lambda_{1},...,\lambda_{p}$ is asymptotically normal, it does not
converge to a Gaussian shift experiment. In particular, the optimality results
available for Gaussian shifts cannot be used in our framework. To analyze
asymptotic risks of various statistical problems related to the experiment of
observing $\lambda_{1},...,\lambda_{p},$ one should directly use Theorem
\ref{AsymptoticNormality}.

In this paper, we use Theorem \ref{AsymptoticNormality} to derive the
asymptotic power envelopes for tests of the null hypothesis $\theta_{0}=0$
against the alternative $\theta_{0}>0.$ Such a power envelope has been derived
by Onatski et al (2013) for the case of PCA. By the Neyman-Pearson lemma, the
most powerful test of the null against a point alternative $\theta_{0}=\theta$
would reject the null when $\ln L\left(  \theta;\Lambda\right)  $ is above a
critical value. By Theorem \ref{AsymptoticNormality} and Le Cam's third lemma
(see van der Vaart (1998), chapter 6),%
\[
\ln L\left(  \theta;\Lambda\right)  \overset{d}{\rightarrow}N\left(  \frac
{1}{4}\ln\left(  1-\left[  \delta\left(  \theta\right)  \right]  ^{2}\right)
,-\frac{1}{2}\ln\left(  1-\left[  \delta\left(  \theta\right)  \right]
^{2}\right)  \right)
\]
under the null, and%
\[
\ln L\left(  \theta;\Lambda\right)  \overset{d}{\rightarrow}N\left(  -\frac
{1}{4}\ln\left(  1-\left[  \delta\left(  \theta\right)  \right]  ^{2}\right)
,-\frac{1}{2}\ln\left(  1-\left[  \delta\left(  \theta\right)  \right]
^{2}\right)  \right)
\]
under the alternative. This implies the following theorem.

\begin{theorem}
Let $\bar{\theta}$ be the threshold corresponding to $F_{\mathbf{\gamma}}$ as
given in Table \ref{Table 3}. For any $\theta\in\left[  0,\bar{\theta}\right)
,$ the value of the asymptotic power envelope for the tests of the null
$\theta_{0}=0$ against the alternative $\theta_{0}>0$ which are based on
$\lambda_{1},...,\lambda_{p}$ and have asymptotic size $\alpha$ is given by%
\[
PE\left(  \theta\right)  =1-\Phi\left[  \Phi^{-1}\left(  1-\alpha\right)
-\sqrt{-\frac{1}{2}\ln\left(  1-\left[  \delta\left(  \theta\right)  \right]
^{2}\right)  }\right]  .
\]
Here $\Phi$ denotes the standard normal cumulative distribution function. For
$\theta\geq\bar{\theta}$ the value of the asymptotic power envelope equals one.
\end{theorem}

The envelopes are different only for the cases that correspond to different
limiting spectral distributions: the semi-circle, the Marchenko-Pastur, and
the Wachter distribution. Therefore, we can denote $PE\left(  \theta\right)  $
as $PE^{SC}\left(  \theta\right)  $ for SMD, as $PE^{MP}\left(  \theta\right)
$ for PCA and REG$_{0}$, and as $PE^{W}\left(  \theta\right)  $ for the
remaining cases. Figure \ref{envelopes} shows the graphs of the envelopes for
$\alpha=0.05$ and $\gamma_{1}=\gamma_{2}=0.9.$ Such large values of
$\gamma_{1}$ and $\gamma_{2}$ correspond to situations where the
dimensionality $p$ is not very different from the \textquotedblleft sample
sizes\textquotedblright\ $n_{1}$ and $n_{2}.$ Of course, the values of
$\gamma_{1}$ and $\gamma_{2}$ are irrelevant for $PE^{SC}\left(
\theta\right)  $, and the value of $\gamma_{2}$ is irrelevant for
$PE^{MP}\left(  \theta\right)  .$%

%TCIMACRO{\FRAME{ftbpFU}{4.2973in}{4.4235in}{0pt}{\Qcb{The asymptotic power
%envelopes $PE^{SC}(\theta),$ $PE^{MP}(\theta),$ and $PE^{W}(\theta)$ for
%$\alpha=0.05,$ $\gamma_{1}=\gamma_{2}=0.9.$}}{\Qlb{envelopes}}{envelopes.eps}%
%{\special{ language "Scientific Word";  type "GRAPHIC";
%maintain-aspect-ratio TRUE;  display "USEDEF";  valid_file "F";
%width 4.2973in;  height 4.4235in;  depth 0pt;  original-width 7.6916in;
%original-height 7.92in;  cropleft "0";  croptop "1";  cropright "1";
%cropbottom "0";  filename 'envelopes.eps';file-properties "XNPEU";}} }%
%BeginExpansion
\begin{figure}[ptb]%
\centering
\includegraphics[
height=4.4235in,
width=4.2973in
]%
{envelopes.eps}%
\caption{The asymptotic power envelopes $PE^{SC}(\theta),$ $PE^{MP}(\theta),$
and $PE^{W}(\theta)$ for $\alpha=0.05,$ $\gamma_{1}=\gamma_{2}=0.9.$}%
\label{envelopes}%
\end{figure}
%EndExpansion


Note that the asymptotic power envelope $PE^{MP}\left(  \theta\right)  $ can
be obtained from $PE^{W}\left(  \theta\right)  $ by sending $\gamma_{2}$ to
zero. Further, $PE^{SC}\left(  \theta\right)  $ can be obtained from
$PE^{MP}\left(  \theta\right)  $ by transformation $\theta\longmapsto
\sqrt{\gamma_{1}}\theta.$ Further, note the difference in the horizontal scale
of the bottom panel of Figure \ref{envelopes} relative to the two other
panels. For $\gamma_{1}=\gamma_{2}=0.9$ the phase transition threshold
corresponding to Wachter distribution is relatively large. It equals $\left(
\gamma_{2}+\rho\right)  /(1-\gamma_{2})\approx18.9$. Moreover, the value of
$PE^{W}\left(  \theta\right)  $ becomes substantially larger than the nominal
size $\alpha=0.05$ for $\theta$ that are situated far below this threshold.
This suggests that the information in all the eigenvalues $\lambda
_{1},...,\lambda_{p}$ might be effectively used to detect spikes that are
small relative to the phase transition threshold in two sample problems. We
leave a confirmation or rejection of this speculation for future research.

\section{Conclusion}

This paper derives the asymptotics of the likelihood ratio processes
corresponding to the null hypothesis of no spikes and the alternative of a
single spike in various high-dimensional multivariate models. We cover all the
five classes of multivariate statistical problems identified by James (1964).
In addition, we consider a symmetric matrix denoising problem that does not
fit in James' classification. We find that, as the dimensionality and the
number of observations go to infinity proportionally, the log likelihood
processes converge to Gaussian limits as long as the value of the spike
parameter is below corresponding phase transition thresholds. We derive
explicit formulae for the autocovariance and the mean of the limiting
processes and use them to obtain asymptotic power envelopes for tests for the
presence of a spike.

In this paper, we make the assumption that $n_{2}\geq p$ to ensure the
invertibility of matrix $E$ in (\ref{basic}) with probability one. However, we
also make the assumption $n_{1}\geq p,$ which can be lifted without a
substantial reformulation of the problem. We make the latter assumption mostly
to simplify our exposition. The assumption is irrelevant for SMD. The PCA
results are obtained in Onatski et al (2013) without using this assumption.
For SigD, our derivations (not reported here) show that the equivalent of
(\ref{LRgeneral}) for $n_{1}<p$ involves the hypergeometric function
$_{2}F_{1}.$ Therefore, SigD with $n_{1}<p$ represents the fifth, rather than
the second, group of multivariate statistical problems according to James'
(1964) classification. The REG$_{0}$ problem is symmetric with respect to
$n_{1}$ and $p$ after a simple reparametrization. For REG, an equivalent of
(\ref{LRgeneral}) for $n_{1}<p$ can be obtained using equation (74) of James
(1964). However, further analysis of REG in this situation needs more
substantial changes to our analysis. In CCA case, the sample canonical
correlations are only well defined if $n_{1}\geq p$. To summarize, when
$n_{1}<p$, the only interesting untreated cases are SigD and REG. We leave
their study for future research.

\section{Appendix}

\subsection{SMD entry of Table \ref{Table 2}}

The explicit expression for $L^{(SMD)}\left(  \theta;\Lambda\right)  $ given
in Table \ref{Table 2} follows from the following lemma.

\begin{lemma}
\label{Lemma 1}For SMD case, the joint density of the diagonal elements of
$\Lambda$ evaluated at the diagonal elements of $x=\operatorname*{diag}%
\left\{  x_{1},...,x_{p}\right\}  $ with $x_{1}\geq...\geq x_{p}$ equals
\begin{equation}
c_{p}\left(  x\right)  \exp\left\{  -p\theta^{2}/4\right\}  \left.  _{0}%
F_{0}\left(  \Psi,x\right)  \right.  , \label{density0}%
\end{equation}
where $c_{p}\left(  x\right)  $ is a quantity that depends on $p$ and $x,$ but
not on $\theta$, and $\Psi=\mathrm{diag}\left\{  \theta p/2,0,...,0\right\}
$. The density under the null hypothesis is obtained from the above expression
by setting $\theta=0$.
\end{lemma}

\textbf{Proof:} The proof is based on the \textquotedblleft symmetrization
trick\textquotedblright\ used by James (1955) to derive the density of
non-central Wishart distribution. Let $Y=U^{\prime}XU,$ where $U$ is a random
matrix from $\mathcal{O(}p\mathcal{)}$ and $X=Z/\sqrt{p}+\eta\theta
\eta^{\prime}$ with $Z$ from GOE, $\theta\geq0,$ and $\left\Vert
\eta\right\Vert =1.$ Note that the eigenvalues of $X$ and $Y$ are the same.
The joint density of the functionally independent elements of $Y$ evaluated at
$y$ is
\[
\left(  2\pi/p\right)  ^{-p(p+1)/4}2^{-p/2}\int_{\mathcal{O(}p\mathcal{)}%
}\operatorname*{etr}\left\{  -\frac{p}{4}\left(  uyu^{\prime}-\eta\theta
\eta^{\prime}\right)  ^{2}\right\}  \mathcal{(}\mathrm{d}u),
\]
where $\mathcal{(}\mathrm{d}u)$ is the normalized uniform measure over
$\mathcal{O(}p\mathcal{)}$. Taking the square under $\operatorname*{etr}$ and
factorizing, we obtain an equivalent expression%
\[
\left(  2\pi/p\right)  ^{-p(p+1)/4}2^{-p/2}\exp\left\{  -\frac{p}{4}\theta
^{2}\right\}  \operatorname*{etr}\left\{  -\frac{p}{4}y^{2}\right\}
\int_{\mathcal{O(}p\mathcal{)}}\operatorname*{etr}\left\{  \frac{p\theta}%
{2}uyu^{\prime}\eta\eta^{\prime}\right\}  \mathcal{(}\mathrm{d}u).
\]
Now change the variables from $y$ to $\left(  H,x\right)  ,$ where
$y=HxH^{\prime}$ is the spectral decomposition of $y,$ and integrate $H$ out
to obtain (\ref{density0}) with
\[
c_{p}\left(  x\right)  =\frac{p^{p(p+1)/4}\pi^{p\left(  p-1\right)  /4}%
}{2^{p\left(  p-1\right)  /4+p}\Gamma_{p}\left(  p/2\right)  }%
\operatorname*{etr}\left(  -\frac{p}{4}x^{2}\right)
%TCIMACRO{\dprod _{i<j}^{p}}%
%BeginExpansion
{\displaystyle\prod_{i<j}^{p}}
%EndExpansion
\left(  x_{i}-x_{j}\right)  .
\]
Here $\Gamma_{p}\left(  p/2\right)  $ is the multivariate Gamma function (see
Muirhead (1982), pp 61-63). $\square$

\subsection{Proof of Lemma \ref{critical}}

It is sufficient to prove the lemma for SigD, REG and CCA. For PCA and
REG$_{0}$, the lemma follows by taking the limits of SigD and REG cases as
$c_{2}\rightarrow0$. For SMD, the lemma then follows by taking the limit of
PCA case as $c_{1}\rightarrow0$, after the transformations $\theta\mapsto
\sqrt{c_{1}}\theta$ and $z\mapsto\sqrt{c_{1}}z+1.$

Our proofs of the lemma are very similar for SigD, REG and CCA. Here we show
only the proof for SigD. First, note that the minimum of $z_{0}$ over
$\theta>0$ equals%
\[
b_{+}\equiv c_{1}\left(  \frac{r+1}{r+c_{2}}\right)  ^{2}%
\]
and is achieved at%
\[
\theta=\bar{\theta}_{p}\equiv\left(  c_{2}+r\right)  /\left(  1-c_{2}\right)
.
\]
Therefore, since $m_{\mathbf{c}}^{W}\left(  z\right)  $ is well defined for
$z>b_{+}$ and since $\bar{\theta}_{p}\rightarrow\bar{\theta}$ as
$\mathbf{n},p\rightarrow_{\mathbf{\gamma}}\infty$, $m_{\mathbf{c}}^{W}\left(
z_{0}\right)  $ must be well-defined for any $\theta\in\left(  0,\bar{\theta
}\right)  $ and for sufficiently large $\mathbf{n},p$. Using an explicit
expression for the Stieltjes transform of the limiting spectral distribution
of the multivariate F matrix, which is given by Bai and Silverstein (2006)
p.71, we obtain%
\begin{align}
m_{\mathbf{c}}^{W}\left(  z\right)   &  =-\frac{\left(  c_{1}-1\right)
\left(  c_{1}-c_{2}z\right)  +\left(  1-c_{2}\right)  c_{1}z}{2c_{1}z\left(
c_{1}-c_{2}z\right)  }\label{stieltjes tilda explicit}\\
&  +\frac{\sqrt{\left(  \left(  c_{1}-c_{2}\right)  z+c_{1}\left(
1-c_{1}\right)  \right)  ^{2}-4c_{1}z\left(  c_{1}-c_{2}z\right)  }}%
{2c_{1}z\left(  c_{1}-c_{2}z\right)  }.\nonumber
\end{align}
Further,
\[
2\frac{\mathrm{d}}{\mathrm{d}z}f_{III}^{(SigD)}(z)=-\frac{\theta r^{2}}%
{c_{1}^{2}\left(  1+\theta\right)  -c_{1}c_{2}\theta z}.
\]
It now takes a direct algebra, which we perform using Maple's symbolic algebra
software, to verify that $z_{0}$ solves equation (\ref{extremum points}%
).$\square$

\subsection{Proof of Lemma \ref{ContourProp1F1}}

For SMD, PCA, and SigD, $\left\vert z-\lambda\right\vert $ is obviously
strictly increasing for any $\lambda\in\mathbb{R}$ and as $z$ moves away from
$z_{0}$ along $\mathcal{K}_{1}$. Therefore,
\[
2\operatorname{Re}f_{II}(z)\equiv\int\ln\left\vert z-\lambda\right\vert
\mathrm{d}F_{\mathbf{c}}\left(  \lambda\right)
\]
is strictly increasing. On the other hand, by (\ref{elementary}),
$\operatorname{Re}f_{III}\left(  z\right)  $ is non-decreasing. Hence
$\operatorname{Re}f\left(  z\right)  $ is strictly increasing.

For REG$_{0}$ and CCA, $\left\vert z-\lambda\right\vert $ is strictly
increasing for any $\lambda\geq0$ as $z$ moves away from $z_{0}$ along
$\mathcal{K}_{1}$ because the center of the circumference that includes
$\mathcal{K}_{1}$ is a negative real number. Therefore, $\operatorname{Re}%
f_{II}(z)$ is strictly increasing. To show that $\operatorname{Re}%
f_{III}\left(  z\right)  $ is strictly increasing too, it is sufficient to
prove that $\operatorname{Re}\varphi_{j}\left(  t_{j}\right)  $ is strictly
increasing for $j=0,2$. A proof of this fact relies on elementary calculus. It
is available from the authors upon request.

For REG, $z$ moves away from $z_{0}$ along $\mathcal{K}_{1}$ when $\tau$ moves
away from $\tau_{0}$ along $\mathcal{C}_{1}$. Using (\ref{phij}), (\ref{f2}),
and (\ref{identity for eta1}), we obtain
\[
\operatorname{Re}f_{III}\left(  \tau\right)  =\frac{1-c_{1}}{c_{1}}\left(
-\operatorname{Re}\tau+\ln\left\vert \tau+1\right\vert +\varepsilon
\ln\left\vert \tau+\varepsilon\right\vert +\varepsilon\ln\varepsilon\right)
.
\]
On the other hand, $\left\vert \tau+\varepsilon\right\vert $ remains constant
on $\mathcal{C}_{1}$ whereas both $-\operatorname{Re}\tau$ and $\left\vert
\tau+1\right\vert $ increase as $\tau$ moves away from $\tau_{0}$ along
$\mathcal{C}_{1}.$ To see that $\left\vert \tau+1\right\vert $ indeed
increases recall that the center $-\varepsilon$ of the circumference that
represents $\mathcal{C}_{1}$ is to the left of the point $-1$. Hence,
$\operatorname{Re}f_{III}\left(  \tau\right)  $ is strictly increasing.

To show that $\operatorname{Re}f_{II}\left(  \tau\right)  $ is strictly
increasing too it is sufficient to verify that%
\[
\left\vert z-\lambda\right\vert \equiv\left\vert \frac{c_{1}\left(
1-c_{1}\right)  }{\theta c_{2}}\frac{\tau\left(  \tau+1\right)  }%
{\tau+\varepsilon}-\lambda\right\vert
\]
is strictly increasing for any $\lambda$ from the support of $F_{\mathbf{c}}.$
Since $\left\vert \tau+\varepsilon\right\vert $ remains constant, it is
sufficient to show that%
\[
\gamma\left(  \tau,x\right)  \equiv\left\vert \tau\left(  \tau+1\right)
-x\left(  \tau+\varepsilon\right)  \right\vert ^{2}%
\]
increases as $\tau$ moves away from $\tau_{0}$ along $\mathcal{C}_{1}$ for any
$x=\lambda\theta c_{2}/\left[  c_{1}\left(  1-c_{1}\right)  \right]  .$

Parameterize $\tau\in\mathcal{C}_{1}$ as $-\varepsilon+\rho e^{\mathrm{i}%
\alpha},\alpha\in\left[  0,\pi/2\right]  .$ Then elementary calculations
yield
\begin{align*}
\gamma\left(  \tau,x\right)   &  =\rho^{4}+\left(  2\varepsilon-1+x\right)
^{2}\rho^{2}-2\rho^{3}\left(  2\varepsilon-1+x\right)  \cos\alpha\\
&  +\varepsilon^{2}\left(  \varepsilon-1\right)  ^{2}+2\left(  \rho^{2}%
\cos2\alpha-\left(  2\varepsilon-1+x\right)  \rho\cos\alpha\right)
\varepsilon\left(  \varepsilon-1\right)
\end{align*}
so that%
\begin{equation}
\frac{\mathrm{d}\gamma\left(  \tau,x\right)  }{\mathrm{d}\cos\alpha}%
=2\rho\left\{  -\left(  2\varepsilon-1+x\right)  \left[  \rho^{2}%
+\varepsilon\left(  \varepsilon-1\right)  \right]  +4\rho\varepsilon\left(
\varepsilon-1\right)  \cos\alpha\right\}  . \label{derivative cosinus1}%
\end{equation}
We would like to prove that the derivative $\mathrm{d}\gamma\left(
\tau,x\right)  /\mathrm{d}\cos\alpha$ is negative. As is seen from
(\ref{derivative cosinus1}), the derivative is decreasing in $x$ and
increasing in $\cos\alpha$. Since $x\geq0$ and $\cos\alpha\leq1$, it is
sufficient to show that $\mathrm{d}\gamma\left(  \tau,0\right)  /\mathrm{d}%
\cos\alpha$ is negative for $\cos\alpha=1.$ We have%
\begin{align*}
\left.  \frac{\mathrm{d}\gamma\left(  \tau,0\right)  }{\mathrm{d}\cos\alpha
}\right\vert _{\cos\alpha=1}  &  =-2\rho\left(  2\varepsilon-1\right)
\left\{  \left(  \rho-\frac{2\varepsilon\left(  \varepsilon-1\right)
}{2\varepsilon-1}\right)  ^{2}+\right. \\
&  \left.  \varepsilon\left(  \varepsilon-1\right)  -\left(  \frac
{2\varepsilon\left(  \varepsilon-1\right)  }{2\varepsilon-1}\right)
^{2}\right\}  .
\end{align*}
This is negative because the expression in the second line of the above
display is positive. Indeed,%
\[
\varepsilon\left(  \varepsilon-1\right)  \left(  2\varepsilon-1\right)
^{2}-4\varepsilon^{2}\left(  \varepsilon-1\right)  ^{2}=\varepsilon\left(
\varepsilon-1\right)  >0.
\]
To summarize, both $\operatorname{Re}f_{II}\left(  \tau\right)  $ and
$\operatorname{Re}f_{III}\left(  \tau\right)  $ are strictly increasing as
$\tau$ moves away from $\tau_{0}$ along $\mathcal{C}_{1}.$ Hence, the image of
$\mathcal{C}_{1},$ $\mathcal{K}_{1},$ is a contour of steep descent of
$-\operatorname{Re}f(z)$ in $z$-plane.

\subsection{Values of $f(z_{0})$ and $\mathrm{d}^{2}f(z_{0})/\mathrm{d}z^{2}$}

Let us first show that $f(z_{0})=0$. Recall that $f(z)=f_{I}+f_{II}%
(z)+f_{III}(z).$ The value of $f_{I}$ is given in Table \ref{Table 2a}. The
value of $f_{III}(z_{0})$ is straightforward to compute using the definitions
of $f_{III}$ and $z_{0}.$ The
\[
2f_{II}\left(  z_{0}\right)  =\int_{b_{-}}^{b_{+}}\ln\left(  z_{0}%
-\lambda\right)  \mathrm{d}F_{\mathbf{c}}\left(  \lambda\right)
\]
takes on three different values: one for SMD, another for PCA and REG$_{0},$
and the third one for SigD, REG, and CCA.

\begin{lemma}
\label{Lemmaf0}For SigD, REG, and CCA, for any $\theta\in\left(  0,\bar
{\theta}\right)  $ and for sufficiently large $\mathbf{n},p,$ we have%
\begin{equation}
2f_{II}\left(  z_{0}\right)  =2\ln c_{1}-\ln\theta-\frac{1-c_{1}}{c_{1}}%
\ln\left(  1+\theta\right)  -\frac{c_{1}+c_{2}}{c_{1}c_{2}}\ln\left(
c_{1}+c_{2}\right)  +\frac{r^{2}}{c_{1}c_{2}}\ln\left[  c_{1}l\left(
\theta\right)  \right]  . \label{formula1}%
\end{equation}

\end{lemma}

\textbf{Proof:} We follow the usual strategy of reduction to a contour
integral. First make the change of variables $\lambda=\alpha-\beta\cos
\varphi.$ In order to arrange that $\lambda=b_{-}$ and $b_{+}$ at $\varphi=0$
and $\pi$ respectively, we set%
\begin{equation}
\alpha=\frac{b_{+}+b_{-}}{2}=\frac{c_{1}\left(  r^{2}+c_{1}^{2}\right)
}{\left(  c_{1}+c_{2}\right)  ^{2}},\qquad\beta=\frac{b_{+}-b_{-}}{2}%
=\frac{2rc_{1}^{2}}{\left(  c_{1}+c_{2}\right)  ^{2}}.\label{alphabeta}%
\end{equation}
We obtain%
\[
2f_{II}\left(  z_{0}\right)  =\frac{c_{1}+c_{2}}{4\pi c_{1}}\int_{0}^{2\pi
}\frac{\beta^{2}\sin^{2}\varphi\ln\left(  z_{0}-\alpha+\beta\cos
\varphi\right)  }{\left(  \alpha-\beta\cos\varphi\right)  \left(  c_{1}%
-c_{2}\alpha+c_{2}\beta\cos\varphi\right)  }\mathrm{d}\varphi
\]
after extending the integral from $\left[  0,\pi\right]  $ to $\left[
0,2\pi\right]  $ using the symmetry of the integrand about $\varphi=\pi$. Now
introduce $z=e^{\mathrm{i}\varphi}.$ Since $\cos\varphi=\left(  z+z^{-1}%
\right)  /2,$ we have from (\ref{alphabeta}) the factorizations%
\begin{align*}
c_{1}\left(  \alpha-\beta\cos\varphi\right)   &  =\frac{\beta}{2r}\left(
r-c_{1}z\right)  \left(  r-c_{1}z^{-1}\right)  ,\\
c_{1}-c_{2}\alpha+c_{2}\beta\cos\varphi &  =\frac{\beta}{2r}\left(
r+c_{2}z\right)  \left(  r+c_{2}z^{-1}\right)  ,\\
z_{0}-\alpha+\beta\cos\varphi &  =q(z)q\left(  z^{-1}\right)  \text{ with}\\
q\left(  z\right)   &  =\frac{c_{1}}{c_{1}+c_{2}}\left(  \sqrt{c_{1}l\left(
\theta\right)  /\theta}+rz\sqrt{\theta/\left[  c_{1}l\left(  \theta\right)
\right]  }\right)  .
\end{align*}
Our integral becomes
\[
2f_{II}\left(  z_{0}\right)  =\frac{-\left(  c_{1}+c_{2}\right)  r^{2}}%
{4\pi\mathrm{i}}\int_{\mathcal{C}}\frac{\left(  z-z^{-1}\right)  ^{2}%
\ln\left(  q(z)q\left(  z^{-1}\right)  \right)  }{\left(  r-c_{1}z\right)
\left(  r-c_{1}z^{-1}\right)  \left(  r+c_{2}z\right)  \left(  r+c_{2}%
z^{-1}\right)  }\frac{\mathrm{d}z}{z}.
\]
The integral has form $I=%
%TCIMACRO{\doint }%
%BeginExpansion
{\displaystyle\oint}
%EndExpansion
\ln\left(  q(z)q\left(  z^{-1}\right)  \right)  H\left(  z\right)
z^{-1}\mathrm{d}z$ with $H(z)=H\left(  z^{-1}\right)  $. Hence, expanding the
logarithm yields two identical terms, so that%
\[
2f_{II}\left(  z_{0}\right)  =\frac{-\left(  c_{1}+c_{2}\right)  }%
{2\pi\mathrm{i}}\int_{\mathcal{C}}\frac{\left(  z^{2}-1\right)  ^{2}\ln
q(z)}{\left(  r-c_{1}z\right)  \left(  z-c_{1}/r\right)  \left(
r+c_{2}z\right)  \left(  z+c_{2}/r\right)  }\frac{\mathrm{d}z}{z}.
\]
For $\theta\in\left(  0,\bar{\theta}\right)  $ and sufficiently large
$\mathbf{n},p,$ we have $\theta\in\left(  0,\bar{\theta}_{p}\right)  $ with
$\bar{\theta}_{p}=\left(  c_{2}+r\right)  /\left(  1-c_{2}\right)  .$ On the
other hand, for $\theta\in\left(  0,\bar{\theta}_{p}\right)  ,$ the function
$\ln q\left(  z\right)  $ is analytic inside the circle $\left\vert
z\right\vert =1,$ and so the whole integrand is analytic inside the circle
except for simple poles at $z=0,c_{1}/r$ and $-c_{2}/r.$ The residues at these
poles are respectively%
\[
\frac{c_{1}+c_{2}}{c_{1}c_{2}}\ln\frac{c_{1}\sqrt{c_{1}l/\theta}}{c_{1}+c_{2}%
},-\frac{1-c_{1}}{c_{1}}\ln\frac{c_{1}\left(  1+\theta\right)  }{\sqrt{\theta
c_{1}l}},\text{ and }-\frac{1-c_{2}}{c_{2}}\ln\frac{c_{1}}{\sqrt{\theta
c_{1}l}}%
\]
and their sum, after collecting terms, yields formula (\ref{formula1}%
).$\square$

\begin{corollary}
\label{Corollaryf0}For PCA and REG$_{0},$ for any $\theta\in\left(
0,\bar{\theta}\right)  $ and for sufficiently large $\mathbf{n},p,$ we have%
\begin{equation}
2f_{II}\left(  z_{0}\right)  =\ln c_{1}-\ln\theta-\frac{1-c_{1}}{c_{1}}%
\ln\left(  1+\theta\right)  +\theta/c_{1}. \label{formula2}%
\end{equation}

\end{corollary}

\textbf{Proof:} The corollary is obtained from Lemma \ref{Lemmaf0} by taking
the limit as $c_{2}\rightarrow0$.$\square$

\begin{corollary}
For SMD, for any $\theta\in\left(  0,\bar{\theta}\right)  $ and for
sufficiently large $\mathbf{n},p,$ we have%
\begin{equation}
2f_{II}\left(  z_{0}\right)  =-\ln\theta+\theta^{2}/2. \label{formula3}%
\end{equation}

\end{corollary}

\textbf{Proof:} We remarked earlier that SMD is a limit of PCA and REG$_{0}$
as $c_{1}\rightarrow0$ after the transformations $\theta\mapsto\sqrt{c_{1}%
}\theta$ and $z\mapsto\sqrt{c_{1}}z+1.$ In particular,%
\[
z_{0}^{(SMD)}=\lim_{c_{1}\rightarrow0,\theta\mapsto\sqrt{c_{1}}\theta}%
(z_{0}^{(PCA)}-1)/\sqrt{c_{1}}\text{ and }F^{SC}\left(  \lambda\right)
=\lim_{c_{1}\rightarrow0}F_{\mathbf{c}}^{MP}\left(  \sqrt{c_{1}}%
\lambda+1\right)  .
\]
These equations imply that%
\[
2f_{II}^{(SMD)}\left(  z_{0}^{(SMD)}\right)  =\lim_{c_{1}\rightarrow
0,\theta\mapsto\sqrt{c_{1}}\theta}\left[  2f_{II}^{(PCA)}\left(  z_{0}%
^{(PCA)}\right)  -\ln\sqrt{c_{1}}\right]  .
\]
Using this relationship together with Corollary \ref{Corollaryf0} yields
$2f_{II}\left(  z_{0}\right)  =-\ln\theta+\theta^{2}/2$ for SMD.$\square$

Combining equations (\ref{formula1}), (\ref{formula2}), and (\ref{formula3})
with the explicit expressions for $f_{I}$ and $f_{III}\left(  z_{0}\right)  $,
we obtain the desired result: $f\left(  z_{0}\right)  =0$ for all the six
cases we consider.

To compute \textrm{d}$^{2}f\left(  z_{0}\right)  /\mathrm{d}z^{2},$ note that
$-2$\textrm{d}$^{2}f_{II}\left(  z_{0}\right)  /\mathrm{d}z^{2}=\mathrm{d}%
m_{\mathbf{c}}\left(  z_{0}\right)  /$\textrm{d}$z.$ Therefore \textrm{d}%
$^{2}f_{II}\left(  z_{0}\right)  /\mathrm{d}z^{2}$ can be directly evaluated
using explicit expressions for the Stieltjes transforms of the semicircle,
Marchenko-Pastur and Wachter distributions. Formula
(\ref{stieltjes tilda explicit}) gives such an explicit expression for
$m_{\mathbf{c}}^{W}\left(  z\right)  .$ The explicit expressions for
$m_{\mathbf{c}}^{MP}\left(  z\right)  $ and $m^{SC}\left(  z\right)  $ are
well known. To perform the necessary computations, we use Maple's symbolic
algebra software. Further, using the definition of $f_{III}(z),$ we directly
evaluate \textrm{d}$^{2}f_{III}\left(  z_{0}\right)  /\mathrm{d}z^{2}.$
Combining the expressions for the second derivatives of $f_{II}$ and
$f_{III},$ we obtain values of the second derivative of $f$ reported in Table
\ref{Table 5}.$\square$

\subsection{Proof of Theorem \ref{Icase}}

First, let us show that%
\begin{equation}
L_{1}\left(  \theta;\Lambda\right)  =\frac{g(z_{0})}{\sqrt{-2\mathrm{d}%
^{2}f(z_{0})/\mathrm{d}z^{2}}}+O_{\mathrm{P}}\left(  p^{-1}\right)  ,
\label{I1case integral}%
\end{equation}
where $O_{\mathrm{P}}\left(  1\right)  $\textit{\ is uniform with rspect to
}$\theta\in\left(  0,\bar{\theta}-\varepsilon\right]  .$ Changing the variable
of integration in (\ref{I1case}) from $z$ to $\zeta=\theta z,$ we obtain%
\begin{equation}
L_{1}\left(  \theta;\Lambda\right)  =\sqrt{\pi p}\frac{1}{2\pi\mathrm{i}}%
\int_{\mathcal{\tilde{K}}}e^{-p\phi(\zeta)}\chi(\zeta)\mathrm{d}\zeta,
\label{olver integral}%
\end{equation}
where
\[
\phi(\zeta)=f\left(  \zeta/\theta\right)  \text{, }\chi(\zeta)=g(\zeta
/\theta)/\theta,
\]
and $\mathcal{\tilde{K}}$ is the image of $\mathcal{K}_{1}\cup\mathcal{\bar
{K}}_{1}$ under the transformation $z\mapsto\zeta.$ The set of possible values
of $\theta$ is $\Omega\equiv\left(  0,\bar{\theta}-\varepsilon\right]  $.

Using Table \ref{Table 5} and the definitions of $\mathcal{K}_{1},$ $z_{0},$
$f(z),$ and $g(z),$ it is straightforward to verify that the assumptions A0-A4
of Lemma \ref{Olver} hold for the integral in (\ref{olver integral}) for all
the six cases that we consider. The validity of A5 follows from Lemma
\ref{gBound} given below and from the definitions of $g\left(  z\right)  $.
Let%
\begin{equation}
\Delta(\zeta)=p\int\ln\left(  \zeta/\theta-\lambda\right)  \mathrm{d}\left(
\hat{F}\left(  \lambda\right)  -F_{\mathbf{c}}\left(  \lambda\right)  \right)
, \label{delta definition}%
\end{equation}
so that $\Delta(\zeta)=-2\ln g_{II}(\zeta/\theta).$

\begin{lemma}
\label{gBound}Suppose that the null hypothesis holds, that is $\theta_{0}%
=0$.\textit{\ Then there exists a positive constant }$C_{1},$ such that for a
subset $\Theta$ of $\mathbb{C}$ that consists of all points whose Euclidean
distance from $\mathcal{\tilde{K}}$ is no larger than $C_{1},$\textit{ we
have}%
\[
\sup_{\zeta\in\Theta}\left\vert \Delta\left(  \zeta\right)  \right\vert
=O_{\mathrm{P}}(1)
\]
\textit{\ as }$\mathbf{n},p\rightarrow_{\mathbf{\gamma}}\infty$\textit{, where
}$O_{\mathrm{P}}(1)$\textit{\ is uniform with respect to }$\theta\in
\Omega\equiv\left(  0,\bar{\theta}-\varepsilon\right]  .$
\end{lemma}

\textbf{Proof:} Let us rewrite (\ref{delta definition}) in the following
equivalent form%
\[
\Delta(\zeta)=p\int\ln\left(  1-\lambda\theta/\zeta\right)  \mathrm{d}\left(
\hat{F}\left(  \lambda\right)  -F_{\mathbf{c}}\left(  \lambda\right)  \right)
.
\]
Statistic $\Delta(\zeta)$ is a special form of a linear spectral statistic
\[
\Delta(\varphi)=p\int\varphi\left(  \lambda\right)  \mathrm{d}\left(  \hat
{F}(\lambda)-F_{\mathbf{c}}\left(  \lambda\right)  \right)
\]
studied by Bai and Yao (2005), Bai and Silverstein (2004), and Zheng (2012)
for the cases of the Semi-circle, Marchenko-Pastur, and Wachter limiting
distributions, respectively. These papers note that%
\[
\Delta(\varphi)=-\frac{p}{2\pi\mathrm{i}}\int_{\mathcal{P}}\varphi\left(
\xi\right)  \left(  \hat{m}\left(  \xi\right)  -m_{\mathbf{c}}\left(
\xi\right)  \right)  \mathrm{d}\xi,
\]
where
\[
\hat{m}\left(  \xi\right)  =\int\frac{1}{\lambda-\xi}\mathrm{d}\hat{F}%
(\xi),\text{ }m_{\mathbf{c}}\left(  \xi\right)  =\int\frac{1}{\lambda-\xi
}\mathrm{d}F_{\mathbf{c}}\left(  \lambda\right)
\]
are the Stieltjes transforms of $\hat{F}$ and $F_{\mathbf{c}},$ and
$\mathcal{P}$ is a positively oriented contour in an open neighborhood of the
supports of $\hat{F}$ and $F_{\mathbf{c}},$ where $\varphi\left(
\zeta\right)  $ is analytic, that encloses these supports. Further, the papers
show that if the distance from $\mathcal{P}$ to the supports of $\hat{F}$ and
$F_{\mathbf{c}}$ stays away from zero with probability approaching one as
$\mathbf{n},p\rightarrow_{\mathbf{\gamma}}\infty$, then%
\begin{equation}
\Delta(\varphi)=-\frac{p}{2\pi\mathrm{i}}\int_{\mathcal{P}}\varphi\left(
\xi\right)  \hat{M}(\xi)\mathrm{d}\xi+O_{\mathrm{P}}\left(  1\right)  ,
\label{modific}%
\end{equation}
where $p\hat{M}(\xi)$ is a truncated version of $p\left[  \hat{m}\left(
\xi\right)  -m_{\mathbf{c}}\left(  \xi\right)  \right]  $ that weakly
converges to a random continuous function on $\mathcal{P}$ with Gaussian
finite dimensional distributions. Furthermore, $O_{\mathrm{P}}\left(
1\right)  $ in (\ref{modific}) is uniform in $\varphi$ that are analytic in
the open neighborhood of the supports of $\hat{F}$ and $F_{\mathbf{c}}$ and
such that $\sup_{\xi\in\mathcal{P}}\left\vert \varphi\left(  \xi\right)
\right\vert <K$ for some constant $K$. Therefore, for any $\delta>0,$ there
exists $B>0,$ such that%
\begin{equation}
\Pr\left(  \left\vert \Delta(\varphi)\right\vert \leq B\sup_{\xi\in
\mathcal{P}}\left\vert \varphi\left(  \xi\right)  \right\vert \right)
>1-\delta\label{boundONdelta}%
\end{equation}
for all $\mathbf{n}$ and $p.$ Moreover, constant $B$ does not depend on
$\varphi$. Now, consider a family of functions $\varphi_{\zeta,\theta}\left(
\xi\right)  $
\[
\left\{  \varphi_{\zeta,\theta}\left(  \xi\right)  =\ln\left(  1-\xi
\theta/\zeta\right)  :\zeta\in\Theta\text{ and }\theta\in\Omega\right\}  .
\]
By the definitions of $\Theta$ and $\Omega,$ there exists an open neighborhood
$\mathcal{N}$ of the supports of $\hat{F}$ and $F_{\mathbf{c}}$ and a constant
$B_{1}$, such that, with probability arbitrarily close to one, for
sufficiently large $\mathbf{n}$ and $p,$ $\varphi_{\zeta,\theta}\left(
\xi\right)  $ are analytic in $\mathcal{N}$ for all $\zeta\in\Theta$ and
$\theta\in\Omega$ and
\[
\sup_{\theta\in\Omega}\sup_{\zeta\in\Theta}\sup_{\xi\in\mathcal{N}}\left\vert
\varphi_{\zeta,\theta}\left(  \xi\right)  \right\vert \leq B_{1}.
\]
Since $\Delta(\varphi_{\zeta,\theta})=\Delta(\zeta)$, we obtain from
(\ref{boundONdelta}) that for any $\delta>0,$ there exists $B_{2}>0$ such that
for sufficiently large $\mathbf{n}$ and $p,$%
\[
\Pr\left(  \sup_{\theta\in\Omega}\sup_{\zeta\in\Theta}\left\vert \Delta
(\zeta)\right\vert \leq B_{2}\right)  >1-\delta.
\]
In other words, $\sup_{\zeta\in\Theta}\left\vert \Delta(\zeta)\right\vert
=O_{\mathrm{P}}(1)$ uniformly over $\theta\in\Omega.\square$

Applying Lemma \ref{Olver} to the integral in (\ref{olver integral}) and using
the fact that $f(z_{0})=0,$ we obtain (\ref{I1case integral}). It remains to
show that $L_{2}\left(  \theta;\Lambda\right)  $ is asymptotically dominated
by $L_{1}\left(  \theta;\Lambda\right)  ,$ where
\[
L_{2}\left(  \theta;\Lambda\right)  =L\left(  \theta;\Lambda\right)
-L_{1}\left(  \theta;\Lambda\right)  .
\]
For SMD, PCA, and SigD we have%
\begin{align*}
\left\vert L_{2}\left(  \theta;\Lambda\right)  \right\vert  &  =\left\vert
\frac{\sqrt{\pi p}}{2\pi\mathrm{i}}\int_{\mathcal{K}_{2}\cup\mathcal{\bar{K}%
}_{2}}e^{-p\left(  f_{I}+f_{III}(z)\right)  }g_{I}g_{III}(z)%
%TCIMACRO{\dprod _{j=1}^{p}}%
%BeginExpansion
{\displaystyle\prod_{j=1}^{p}}
%EndExpansion
\left(  z-\lambda_{j}\right)  ^{-1/2}\mathrm{d}z\right\vert \\
&  \leq\sqrt{\frac{p}{\pi}}e^{-pf_{I}}g_{I}\left(  2z_{0}\right)  ^{-p/2}%
\int_{\mathcal{K}_{2}}\left\vert e^{-pf_{III}(z)}g_{III}(z)\mathrm{d}%
z\right\vert \\
&  \leq\sqrt{\frac{p}{\pi}}e^{-pf_{I}}g_{I}\left(  2z_{0}\right)  ^{-p/2}%
\int_{-\infty}^{z_{0}}e^{-pf_{III}(x)}g_{III}(x)\mathrm{d}x.
\end{align*}
Explicitly evaluating the latter integral and using the exact form of $g_{I}$,
available from Table \ref{Table 2a}, we obtain%
\[
\left\vert L_{2}\left(  \theta;\Lambda\right)  \right\vert \leq\frac{2C}%
{\sqrt{\pi p}}e^{-pf_{I}}\left(  2z_{0}\right)  ^{-p/2}e^{-pf_{III}(z_{0}%
)}\left(  1+o(1)\right)  ,
\]
where $o(1)$ does not depend on $\theta$, $C=1$ for SMD and PCA, and
$C=\sqrt{c_{1}+c_{2}}/r$ for SigD. Therefore,%
\begin{align*}
\left\vert L_{2}\left(  \theta;\Lambda\right)  \right\vert  &  \leq\frac
{2C}{\sqrt{\pi p}}e^{-pf\left(  z_{0}\right)  }\exp\left\{  -p\left(
\ln(2z_{0})/2-f_{II}\left(  z_{0}\right)  \right)  \right\}  \left(
1+o(1)\right) \\
&  =\frac{2C}{\sqrt{\pi p}}\exp\left\{  -\frac{p}{2}\int\ln\left(
\frac{2z_{0}}{z_{0}-\lambda}\right)  \mathrm{d}F_{\mathbf{c}}(\lambda
)\right\}  \left(  1+o(1)\right)  ,
\end{align*}
where we used the fact that $f\left(  z_{0}\right)  =0$. But $\ln\left(
2z_{0}/\left(  z_{0}-\lambda\right)  \right)  $ is positive and bounded away
from zero uniformly over $\theta\in\left(  0,\bar{\theta}-\varepsilon\right]
$ with probability arbitrarily close to one, for sufficiently large
$\mathbf{n},p$. Hence, there exists a positive constant $K$ such that%
\[
\left\vert L_{2}\left(  \theta;\Lambda\right)  \right\vert \leq\frac{2C}%
{\sqrt{\pi p}}e^{-pK}\left(  1+o(1)\right)
\]
with probability arbitrarily close to one for sufficiently large
$\mathbf{n},p$. Combining this inequality with (\ref{I1case integral}), we
establish Theorem \ref{Icase} for SMD, PCA, and SigD.

For REG$_{0}$, we shall need the following lemma.

\begin{lemma}
For sufficiently large $\mathbf{n}$ and $p$, we have%
\begin{equation}
\left\vert \left.  _{0}F_{1}\left(  b-s;\Psi_{11}z\right)  \right.
\right\vert <4\sqrt{\pi m}\left\vert \exp\left\{  -m\varphi_{0}(t_{0}%
)\right\}  \right\vert \label{Bessel inequality}%
\end{equation}
for any $z$ and any $\theta>0$.
\end{lemma}

\textbf{Proof:} We use the identity (see formula 9.6.3 in Abramowitz and
Stegun (1964))%
\[
I_{m}\left(  \zeta\right)  =e^{-m\pi\mathrm{i}/2}J_{m}\left(  \mathrm{i}%
\zeta\right)  \text{ for }-\pi<\arg\zeta\leq\pi/2,
\]
where $J_{m}\left(  \cdot\right)  $ is the Bessel function. The identity and
(\ref{bessel rep}) imply that%
\begin{equation}
\left.  _{0}F_{1}\left(  b-s;\Psi_{11}z\right)  \right.  =\Gamma\left(
m+1\right)  \left(  m^{2}\eta_{0}\right)  ^{-m/2}e^{-m\pi\mathrm{i}/2}%
J_{m}\left(  \mathrm{i}2m\eta_{0}^{1/2}\right)  . \label{new bessel}%
\end{equation}
On the other hand, for any $\zeta$ and any positive $K$,
\begin{equation}
\left\vert J_{K}\left(  K\zeta\right)  \right\vert \leq\left\{  1+\left\vert
\frac{\sin K\pi}{K\pi}\right\vert \right\}  \left\vert \left\{  \frac
{\zeta\exp\left\{  \sqrt{1-\zeta^{2}}\right\}  }{1+\sqrt{1-\zeta^{2}}%
}\right\}  ^{K}\right\vert , \label{Watson inequality}%
\end{equation}
(see Watson (1944), p.270). The latter inequality, equation (\ref{new bessel}%
), and the Stirling formula for $\Gamma\left(  m+1\right)  $ imply that
(\ref{Bessel inequality}) holds for sufficiently large $m$, for any $z$ and
$\theta>0$. The constant $4$ on the right hand side of
(\ref{Bessel inequality}) is not the smallest possible one, but it is
sufficient for our purposes.$\square$

Using inequality (\ref{Bessel inequality}), we obtain for REG$_{0}$%
\begin{equation}
\left\vert L_{2}\left(  \theta;\Lambda\right)  \right\vert \leq4e^{-pf_{I}%
}g_{I}\sqrt{pm}\int_{\mathcal{K}_{2}}\left\vert \exp\left\{  -m\varphi
_{0}(t_{0})\right\}
%TCIMACRO{\dprod _{j=1}^{p}}%
%BeginExpansion
{\displaystyle\prod_{j=1}^{p}}
%EndExpansion
\left(  z-\lambda_{j}\right)  ^{-1/2}\mathrm{d}z\right\vert .
\label{first additional}%
\end{equation}
It is straightforward to verify that $\operatorname{Re}\varphi_{0}(t_{0})$ is
strictly increasing as $z$ is moving along $\mathcal{K}_{2}$ towards $-\infty
$. Therefore, for any $z\in\mathcal{K}_{2},$
\[
\operatorname{Re}\varphi_{0}\left(  t_{0}(z)\right)  >\operatorname{Re}%
\varphi_{0}(t_{0}(\bar{z})),
\]
where $\bar{z}=z_{1}+\mathrm{i}\left(  z_{0}-z_{1}\right)  $ is the point of
$\mathcal{K}_{2}$ where $\mathcal{K}_{2}$ meets $\mathcal{K}_{1}.$ The latter
inequality together with (\ref{first additional}) yields%
\[
\left\vert L_{2}\left(  \theta;\Lambda\right)  \right\vert \leq
4e^{-p\operatorname{Re}f(\bar{z})}g_{I}\left\vert g_{II}\left(  \bar
{z}\right)  \right\vert \sqrt{pm}\int_{\mathcal{K}_{2}}%
%TCIMACRO{\dprod _{j=1}^{p}}%
%BeginExpansion
{\displaystyle\prod_{j=1}^{p}}
%EndExpansion
\left\vert \frac{\bar{z}-\lambda_{j}}{z-\lambda_{j}}\right\vert ^{1/2}%
\left\vert \mathrm{d}z\right\vert .
\]
Since, for some constant $\tau_{1}$, $\operatorname{Re}f\left(  \bar
{z}\right)  >f\left(  z_{0}\right)  +\tau_{1}=\tau_{1}$ and since, by Lemma
\ref{gBound}, $4g_{II}\left(  \bar{z}\right)  =O_{\mathrm{P}}\left(  1\right)
$ uniformly over $\theta\in\left(  0,\bar{\theta}-\varepsilon\right]  ,$ we
obtain%
\begin{equation}
\left\vert L_{2}\left(  \theta;\Lambda\right)  \right\vert \leq e^{-p\tau_{1}%
}g_{I}\sqrt{pm}\int_{\mathcal{K}_{2}}\prod_{j=1}^{p}\left\vert \frac{\bar
{z}-\lambda_{j}}{z-\lambda_{j}}\right\vert ^{1/2}\left\vert \mathrm{d}%
z\right\vert O_{\mathrm{P}}\left(  1\right)  . \label{to combine with}%
\end{equation}
Note that for any $z\in\mathcal{K}_{2}$ and any $j=1,...,p,$ $\left\vert
\left(  \bar{z}-\lambda_{j}\right)  /\left(  z-\lambda_{j}\right)  \right\vert
\leq1$ and $\left\vert z-\lambda_{j}\right\vert >\left\vert z\right\vert .$
Further, since $z_{0}<\left\vert \bar{z}\right\vert $ and with probability
arbitrary close to one, for sufficiently large $\mathbf{n}$ and $p,$
$\lambda_{1}<z_{0},$ we have $\left\vert \bar{z}-\lambda_{j}\right\vert
<\left\vert \bar{z}-z_{0}\right\vert <2\left\vert \bar{z}\right\vert .$ Thus,
for $p\geq4,$ we have%
\[
\int_{\mathcal{K}_{2}}\prod_{j=1}^{p}\left\vert \frac{\bar{z}-\lambda_{j}%
}{z-\lambda_{j}}\right\vert ^{1/2}\left\vert \mathrm{d}z\right\vert \leq
\int_{\mathcal{K}_{2}}4\left\vert z/\bar{z}\right\vert ^{-2}\left\vert
\mathrm{d}z\right\vert =\left\vert \bar{z}\right\vert O(1)
\]
Combining this with (\ref{to combine with}) and noting that $g_{I}\left\vert
\bar{z}\right\vert =O\left(  1\right)  $ uniformly over $\theta\in\left(
0,\bar{\theta}-\varepsilon\right]  ,$ we obtain
\begin{equation}
\left\vert L_{2}\left(  \theta;\Lambda\right)  \right\vert \leq\sqrt
{pm}e^{-p\tau_{1}}O_{\mathrm{P}}\left(  1\right)  , \label{REG0inequality}%
\end{equation}
where $O_{\mathrm{P}}\left(  1\right)  $ is uniform with respect to $\theta
\in\left(  0,\bar{\theta}-\varepsilon\right]  $. Theorem \ref{Icase} for
REG$_{0}$ follows from the latter equality and (\ref{I1case integral}).

For REG and CCA, the Theorem follows from (\ref{I1case integral}) and
inequalities%
\begin{equation}
\left\vert L_{2}\left(  \theta;\Lambda\right)  \right\vert \leq pe^{-p\tau
_{2}}O_{\mathrm{P}}\left(  1\right)  , \label{restInequalities}%
\end{equation}
where $\tau_{2}$ is a positive constant. We obtain (\ref{restInequalities}) by
combining the method used to derive (\ref{REG0inequality}) with upper bounds
on $_{1}F_{1}$ and $_{2}F_{1},$ which we establish using the integral
representations (\ref{integralrepF11}). We omit details to save space.$\square
$

\begin{thebibliography}{99}                                                                                               %


\bibitem {ASt}Abramowitz, M., and Stegun, I. A. (eds.) (1964) \textit{Handbook
of Mathematical Functions with Formulas, Graphs, and Mathematical Tables},
National Bureau of Standards.

\bibitem {Bai}Bai, Z.D. (1999) \textquotedblleft Methodologies in Spectral
Analysis of Large Dimensional Random Matrices, a Review,\textquotedblright%
\ \textit{Statistica Sinica }9, 611-677.

\bibitem {BS}Bai, Z.D. and J.W. Silverstein (2004) \textquotedblleft CLT for
Linear Spectral Statistics of Large-Dimensional Sample Covariance
Matrices\textquotedblright, \textit{Annals of Probability} 32, 553-605.

\bibitem {BS2006}Bai, Z.D., and J.W. Silverstein (2006) \textit{Spectral
Analysis of Large Dimensional Random Matrices}, Science Press, Beijing.

\bibitem {BaikS}Baik, J. and J.W. Silverstein. (2006) \textquotedblleft
Eigenvalues of large sample covariance matrices of spiked population
models\textquotedblright. Journal of Multivariate Analysis 97, 1382--1408.

\bibitem {BY}Bai, Z.D., and Yao, J. (2005) \textquotedblleft On the
Convergence of the Spectral Empirical Process of Wigner
matrices\textquotedblright, Bernoulli 11(6), 1059-1092.

\bibitem {Bao}Bao, Z., J. Hu, G. Pan, and W. Zhou (2014) \textquotedblleft
Canonical correlation coefficients of high-dimensional normal vectors: finite
rank case,\textquotedblright\ arXiv 1407.7194

\bibitem {DJ}Dharmawansa, P. and I. M. Johnstone (2014) \textquotedblleft
Joint density of eigenvalues in spiked multivariate models,\textquotedblright%
\ Stat 3, no. 1, 240--249.

\bibitem {DJO}Dharmawansa, P., I.M. Johnstone, and A. Onatski (2014a)
\textquotedblleft Local Asymptotic Normality of the spectrum of
high-dimensional spiked F-ratios,\textquotedblright\ arXiv 1411.3875

\bibitem {Fyod}Fyodorov, Y. V., B.A. Khoruzhenko, and N.J. Simm (2013)
\textquotedblleft Fractional Brownian motion with Hurst index H=0 and the
Gaussian Unitary Ensemble,\textquotedblright\ arXiv 1312.0212v1.

\bibitem {James1955}James, A.T. (1955) \textquotedblleft The Non-central
Wishart Distribution,\textquotedblright\ \textit{Proceedings of the Royal
Society of London. Series A, Mathematical and Physical Sciences} 229, pp. 364-366.

\bibitem {James}James, A. T. (1964) \textquotedblleft Distributions of matrix
variates and latent roots derived from normal samples,\textquotedblright%
\ \ \textit{Annals of Mathematical Statistics} 35, 475-501.

\bibitem {L}Luke, Y.L. (1969) \textit{The Special Functions and Their
Approximations}, Volume I. Academic Press.

\bibitem {Mai}Ma\"{\i}da, M. (2007) \textquotedblleft Large deviations for the
largest eigenvalue of rank one deformations of Gaussian
ensembles,\textquotedblright\ Electronic Journal of Probability 12, 1131--1150.

\bibitem {Mehta}Mehta, L.M. (2004) \textit{Random Matrices}. Third edition.
Elsevier Academic Press.

\bibitem {Mon}Montanari, A., Reichman, D., and Zeitouni, O. (2014)
\textquotedblleft On the limitation of spectral methods: From the Gaussian
hidden clique problem to rank one perturbations of Gaussian
tensors,\textquotedblright\ arXiv: 1411.6149v1.

\bibitem {Mu}Muirhead, R.J. (1978) \textquotedblleft Latent roots and matrix
variates: a review of some asymptotic results,\textquotedblright%
\ \textit{Annals of Statistics} 6, 5-33.

\bibitem {Muirhead}Muirhead, R.J. (1982) Aspects of Multivariate Statistical
Theory. John Wiley \& Son, Hoboken, New Jersey.

\bibitem {NS}Nadakuditi, R.R. and J. W. Silverstein (2010) \textquotedblleft
Fundamental Limit of Sample Generalized Eigenvalue Based Detection of Signals
in Noise Using Relatively Few Signal-Bearing and Noise-Only
Samples,\textquotedblright\ IEEE Journal of Selected Topics in Signal
Processing 4 (3), 468-480.

\bibitem {OL97}Olver, F.W.J. (1997) \textit{Asymptotics and Special
Functions}, A K Peters, Natick, Massachusetts.

\bibitem {Oletal}Olver, F.W.J., D. W. Lozier, R.F.Boisvert, and C.W. Clark
(eds.) (2010) \textit{NIST Handbook of Mathematical Functions}, Cambridge
University Press, Cambridge.

\bibitem {O0}Onatski, A. (2007) \textquotedblleft Asymptotics of the principal
components estimator of large factor models with weak factors and i.i.d.
Gaussian noise,\textquotedblright\ manuscript, University of Cambridge.
Available at http://www.econ.cam.ac.uk /people /faculty /ao319 /pubs /inference45a.pdf.

\bibitem {OMH}Onatski, A., Moreira, M.J., and Hallin, M. (2013)
\textquotedblleft Asymptotic power of sphericity tests for high-dimensional
data\textquotedblright, \textit{Annals of Statistics} 41, 1204-1231.

\bibitem {P0}Paris, R.B. (2013a) \textquotedblleft Asymptotics of the Gauss
Hypergeometric Function with Large Parameters, I,\textquotedblright\ Journal
of Classical Analysis 2, 183-203.

\bibitem {P2}Paris, R.B. (2013b) \textquotedblleft Asymptotics of the Gauss
Hypergeometric Function with Large Parameters, II,\textquotedblright\ Journal
of Classical Analysis 3, 1-15.

\bibitem {Pass}Passemier, D., McKay, M.R., and Y. Chen (2014)
\textquotedblleft Asymptotic Linear Spectral Statistics for Spiked Hermitian
Random Matrix Models\textquotedblright, arXiv:1402.6419v1.

\bibitem {Silverstein}Silverstein, J.W. (1985) \textquotedblleft The Limiting
Eigenvalue Distribution of a Multivariate F Matrix,\textquotedblright%
\ \textit{SIAM Journal of Mathematical Analysis} 16 (3), 641-646.

\bibitem {vanderVaart}van der Vaart, A.W. (1998) \textit{Asymptotic
Statistics}, Cambridge University Press.

\bibitem {Wachter}Wachter, K. (1980) \textquotedblleft The limiting empirical
measure of multiple discriminant ratios,\textquotedblright\ \textit{The Annals
of Statistics} 8, 937-957.

\bibitem {W}Watson, G.N. (1944) \textit{A Treatise on the Theory of Bessel
Functions}, Cambridge University Press.

\bibitem {YBK}Yin, Y.Q, Bai, Z.D., and Krishnaiah, P.R. (1983)
\textquotedblleft Limiting Behavior of the Eigenvalues of a Multivariate F
Matrix,\textquotedblright\ \textit{Journal of Multivariate Analysis} 13 (4), 508-516.

\bibitem {Zheng}Zheng, S. (2012) \textquotedblleft Central limit theorems for
linear spectral statistics of large dimensional F-matrices\textquotedblright,
\textit{Annales de l'Institut Henry Poincar\'{e} - Probabilit\'{e}s et
Statistiques} 48, 444-476.
\end{thebibliography}


\end{document}
1509.07269.extracted_bib
{AStAbramowitz, M., and Stegun, I. A. (eds.) (1964) Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, National Bureau of Standards.
1509.07269.extracted_bib
{BaiBai, Z.D. (1999) \textquotedblleft Methodologies in Spectral Analysis of Large Dimensional Random Matrices, a Review,\textquotedblright% \ Statistica Sinica 9, 611-677.
1509.07269.extracted_bib
{BSBai, Z.D. and J.W. Silverstein (2004) \textquotedblleft CLT for Linear Spectral Statistics of Large-Dimensional Sample Covariance Matrices\textquotedblright, Annals of Probability 32, 553-605.
1509.07269.extracted_bib
{BS2006Bai, Z.D., and J.W. Silverstein (2006) Spectral Analysis of Large Dimensional Random Matrices, Science Press, Beijing.
1509.07269.extracted_bib
{BaikSBaik, J. and J.W. Silverstein. (2006) \textquotedblleft Eigenvalues of large sample covariance matrices of spiked population models\textquotedblright. Journal of Multivariate Analysis 97, 1382--1408.
1509.07269.extracted_bib
{BYBai, Z.D., and Yao, J. (2005) \textquotedblleft On the Convergence of the Spectral Empirical Process of Wigner matrices\textquotedblright, Bernoulli 11(6), 1059-1092.
1509.07269.extracted_bib
{BaoBao, Z., J. Hu, G. Pan, and W. Zhou (2014) \textquotedblleft Canonical correlation coefficients of high-dimensional normal vectors: finite rank case,\textquotedblright\ arXiv 1407.7194
1509.07269.extracted_bib
{DJDharmawansa, P. and I. M. Johnstone (2014) \textquotedblleft Joint density of eigenvalues in spiked multivariate models,\textquotedblright% \ Stat 3, no. 1, 240--249.
1509.07269.extracted_bib
{DJODharmawansa, P., I.M. Johnstone, and A. Onatski (2014a) \textquotedblleft Local Asymptotic Normality of the spectrum of high-dimensional spiked F-ratios,\textquotedblright\ arXiv 1411.3875
1509.07269.extracted_bib
{FyodFyodorov, Y. V., B.A. Khoruzhenko, and N.J. Simm (2013) \textquotedblleft Fractional Brownian motion with Hurst index H=0 and the Gaussian Unitary Ensemble,\textquotedblright\ arXiv 1312.0212v1.
1509.07269.extracted_bib
{James1955James, A.T. (1955) \textquotedblleft The Non-central Wishart Distribution,\textquotedblright\ Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences 229, pp. 364-366.
1509.07269.extracted_bib
{JamesJames, A. T. (1964) \textquotedblleft Distributions of matrix variates and latent roots derived from normal samples,\textquotedblright% \ \ Annals of Mathematical Statistics 35, 475-501.
1509.07269.extracted_bib
{LLuke, Y.L. (1969) The Special Functions and Their Approximations, Volume I. Academic Press.
1509.07269.extracted_bib
{MaiMa\"{\ida, M. (2007) \textquotedblleft Large deviations for the largest eigenvalue of rank one deformations of Gaussian ensembles,\textquotedblright\ Electronic Journal of Probability 12, 1131--1150.
1509.07269.extracted_bib
{MehtaMehta, L.M. (2004) Random Matrices. Third edition. Elsevier Academic Press.
1509.07269.extracted_bib
{MonMontanari, A., Reichman, D., and Zeitouni, O. (2014) \textquotedblleft On the limitation of spectral methods: From the Gaussian hidden clique problem to rank one perturbations of Gaussian tensors,\textquotedblright\ arXiv: 1411.6149v1.
1509.07269.extracted_bib
{MuMuirhead, R.J. (1978) \textquotedblleft Latent roots and matrix variates: a review of some asymptotic results,\textquotedblright% \ Annals of Statistics 6, 5-33.
1509.07269.extracted_bib
{MuirheadMuirhead, R.J. (1982) Aspects of Multivariate Statistical Theory. John Wiley \& Son, Hoboken, New Jersey.
1509.07269.extracted_bib
{NSNadakuditi, R.R. and J. W. Silverstein (2010) \textquotedblleft Fundamental Limit of Sample Generalized Eigenvalue Based Detection of Signals in Noise Using Relatively Few Signal-Bearing and Noise-Only Samples,\textquotedblright\ IEEE Journal of Selected Topics in Signal Processing 4 (3), 468-480.
1509.07269.extracted_bib
{OL97Olver, F.W.J. (1997) Asymptotics and Special Functions, A K Peters, Natick, Massachusetts.
1509.07269.extracted_bib
{OletalOlver, F.W.J., D. W. Lozier, R.F.Boisvert, and C.W. Clark (eds.) (2010) NIST Handbook of Mathematical Functions, Cambridge University Press, Cambridge.
1509.07269.extracted_bib
{O0Onatski, A. (2007) \textquotedblleft Asymptotics of the principal components estimator of large factor models with weak factors and i.i.d. Gaussian noise,\textquotedblright\ manuscript, University of Cambridge. Available at http://www.econ.cam.ac.uk /people /faculty /ao319 /pubs /inference45a.pdf.
1509.07269.extracted_bib
{OMHOnatski, A., Moreira, M.J., and Hallin, M. (2013) \textquotedblleft Asymptotic power of sphericity tests for high-dimensional data\textquotedblright, Annals of Statistics 41, 1204-1231.
1509.07269.extracted_bib
{P0Paris, R.B. (2013a) \textquotedblleft Asymptotics of the Gauss Hypergeometric Function with Large Parameters, I,\textquotedblright\ Journal of Classical Analysis 2, 183-203.
1509.07269.extracted_bib
{P2Paris, R.B. (2013b) \textquotedblleft Asymptotics of the Gauss Hypergeometric Function with Large Parameters, II,\textquotedblright\ Journal of Classical Analysis 3, 1-15.
1509.07269.extracted_bib
{PassPassemier, D., McKay, M.R., and Y. Chen (2014) \textquotedblleft Asymptotic Linear Spectral Statistics for Spiked Hermitian Random Matrix Models\textquotedblright, arXiv:1402.6419v1.
1509.07269.extracted_bib
{SilversteinSilverstein, J.W. (1985) \textquotedblleft The Limiting Eigenvalue Distribution of a Multivariate F Matrix,\textquotedblright% \ SIAM Journal of Mathematical Analysis 16 (3), 641-646.
1509.07269.extracted_bib
{vanderVaartvan der Vaart, A.W. (1998) Asymptotic Statistics, Cambridge University Press.
1509.07269.extracted_bib
{WachterWachter, K. (1980) \textquotedblleft The limiting empirical measure of multiple discriminant ratios,\textquotedblright\ The Annals of Statistics 8, 937-957.
1509.07269.extracted_bib
{WWatson, G.N. (1944) A Treatise on the Theory of Bessel Functions, Cambridge University Press.
1509.07269.extracted_bib
{YBKYin, Y.Q, Bai, Z.D., and Krishnaiah, P.R. (1983) \textquotedblleft Limiting Behavior of the Eigenvalues of a Multivariate F Matrix,\textquotedblright\ Journal of Multivariate Analysis 13 (4), 508-516.
1509.07269.extracted_bib
{ZhengZheng, S. (2012) \textquotedblleft Central limit theorems for linear spectral statistics of large dimensional F-matrices\textquotedblright, Annales de l'Institut Henry Poincar\'{e - Probabilit\'{es et Statistiques 48, 444-476.