Is Q-Learning Minimax Optimal? \\ A Tight Sample Complexity Analysis
Gen Li\thanks{Department of Electrical and Computer Engineering
Princeton University
Princeton
NJ 08544
USA.
Q-learning, which seeks to learn the optimal Q-function of a Markov decision process (MDP) in a model-free fashion, lies at the heart of reinforcement learning. When it comes to the synchronous setting (such that independent samples for all state-action pairs are drawn from a generative model in each iteration), substantial progress has been made towards understanding the sample efficiency of Q-learning. Consider a $\gamma$-discounted infinite-horizon MDP with state space $\cS$ and action space $\cA$: to yield an entrywise $\varepsilon$-approximation of the optimal Q-function, state-of-the-art theory for Q-learning requires a sample size exceeding the order of $\frac{|\mathcal{S||\mathcal{A|{(1-\gamma)^5\varepsilon^{2$, which fails to match existing minimax lower bounds. This gives rise to natural questions: what is the sharp sample complexity of Q-learning? Is Q-learning provably sub-optimal? This paper addresses these questions for the synchronous setting: (1) when $|\cA|=1$ (so that Q-learning reduces to TD learning), we prove that the sample complexity of TD learning is minimax optimal and scales as $\frac{|\mathcal{S|{(1-\gamma)^3\varepsilon^2$ (up to log factor); (2) when $|\cA|\geq 2$, we settle the sample complexity of Q-learning to be on the order of $\frac{|\mathcal{S||\mathcal{A|{(1-\gamma)^4\varepsilon^2$ (up to log factor). Our theory unveils the strict sub-optimality of Q-learning when $|\cA|\geq 2$, and rigorizes the negative impact of over-estimation in Q-learning. Finally, we extend our analysis to accommodate asynchronous Q-learning (i.e., the case with Markovian samples), sharpening the horizon dependency of its sample complexity to be $\frac{1{(1-\gamma)^4$.
AsynQ.tex
\section{Extension: sample complexity of asynchronous Q-learning}
\label{sec:AsynQ}


Moving beyond the synchronous setting, another scenario of practical importance is the case where the acquired samples take the form of a single Markovian trajectory \citep{tsitsiklis1994asynchronous}.  
In this section, we extend our analysis framework for synchronous Q-learning to accommodate Markovian non-i.i.d.~samples.   


\subsection{Markovian samples and asynchronous Q-learning}


\paragraph{Markovian sample trajectory.} Suppose that we obtain a Markovian sample trajectory $\{(s_t, a_t,r_t)\}_{t=0}^{\infty}$, which is generated by the MDP of interest when a stationary behavior policy $\pib$ is employed; in other words, 
%
\begin{equation}
	a_t \sim \pib(\cdot\mymid s_t),  \quad r_t = r(s_t,a_t),\quad s_{t+1} \sim P(\cdot\mymid s_t, a_t),\qquad t\geq 0. 
\end{equation}
%
When $\pib$ is stationary, the trajectory $\{(s_t,a_t)\}_{t=0}^{\infty}$ can be viewed as a sample path of a time-homogeneous Markov chain; in what follows, we shall denote by $\mu_{\pib}$ the stationary distribution of this Markov chain.  
Note that the behavior policy $\pib$ can often be quite different from the target optimal policy $\pi^{\star}$. 


\paragraph{Asynchronous Q-learning.} 

In the presence of a single Markovian sample trajectory, the Q-learning algorithm implements the following iterative update rule
%
\begin{subequations}
\label{eq:async-Q-update-rule}
\begin{align}
Q_{t}(s_{t-1},a_{t-1}) & =(1-\eta_{t})Q_{t-1}(s_{t-1},a_{t-1})+\eta_{t}\Big\{ r(s_{t-1},a_{t-1})+\gamma\max_{a'\in\mathcal{A}}Q_{t-1}(s_{t},a')\Big\} , \\
Q_{t}(s,a) & =Q_{t-1}(s,a)\qquad\text{for all }(s,a)\neq(s_{t-1},a_{t-1})
\end{align}
\end{subequations}
%
for all $t\geq 1$, where $0<\eta_t\leq 1$ stands for the learning rate at time $t$. It is often referred to as {\em asynchronous Q-learning}, as only a single state-action pair is updated in each iteration (in contrast, synchronous Q-learning updates all state-action pairs simultaneously in each iteration). This also leads to the following estimate for the value function at time $t$:
%
\begin{align}
	V_t(s) \coloneqq \max_{a\in \cA} Q_t(s,a) \qquad \text{for all }s\in \mathcal{S}.
	\label{eq:Vt-async-Q-defn}
\end{align}
%
As can be expected, the presence of Markovian non-i.i.d.~data considerably complicates the analysis for asynchronous Q-learning. 


\paragraph{Assumptions.}
In order to ensure sufficient coverage of the sample trajectory over the state/action space, we make the following assumption throughout this section, which is also commonly imposed in prior literature. 
%
\begin{assumption}
	The Markov chain induced by the behavior policy $\pib$ is uniformly ergodic.\footnote{See \citet[Section~1.2]{paulin2015concentration} for the definition of uniform ergodicity.} 
\end{assumption}
%
In addition, there are two crucial quantities concerning the sample trajectory that dictate the performance of asynchronous Q-learning. The first one is the minimum state-action occupancy probability of the sample trajectory, defined formally as 
%
\begin{equation}
	\label{defn:mu-min}
	\mumin \defn \min_{(s,a)\in \cS\times \cA} \mu_{\pib}(s,a). 
\end{equation}
%
This metric captures the information bottleneck incurred by the least visited state-action pair. The second key quantity is the mixing time associated with the sample trajectory, denoted by
%
\begin{equation}
	\label{defn-tmix}
	\tmix 
	\defn \min \Big\{ t \,\Big| \max_{(s,a)\in \cS\times \cA} d_{\mathsf{TV}}\big( P^t(\cdot\mymid s,a), \mu_{\pib} \big) \leq \frac{1}{4} \Big\}.
\end{equation}
%
Here, $d_{\mathsf{TV}}(\mu,\nu)\coloneqq \frac{1}{2}\sum_{x\in \mathcal{X}} |\mu(x) - \nu(x)|$ indicates the total variation distance between two measures $\mu$ and $\nu$ over $\mathcal{X}$ \citep{tsybakov2009introduction}, whereas $P^t(\cdot\mymid s,a)$ stands for the distribution of $(s_t,a_t)$ when the sample trajectory is initialized at $(s_0,a_0)=(s,a)$. 
In words, the mixing time reflects the time required for the Markov chain to become nearly independent of the initial states. See \citet[Section 2]{li2020sample} for a more detailed account of these quantities and assumptions. 




\subsection{Sample complexity of asynchronous Q-learning}


While a number of previous works have been dedicated to understanding the performance of asynchronous Q-learning, 
its sample complexity bound remains loose when it comes to the dependency on the effective horizon $\frac{1}{1-\gamma}$. 
Encouragingly, the analysis framework laid out in this paper allows us to tighten the dependency on $\frac{1}{1-\gamma}$, 
as stated below. 
% 
\begin{theorem}
\label{thm:infinite-horizon-asyn}
%
Consider any $\delta\in(0,1)$,  $\varepsilon\in(0,1]$, and $\gamma\in [1/2,1)$. Suppose that for any $0\leq t\leq T$, the
learning rates satisfy
%
\begin{subequations}
\label{eq:thm-infinite-horizon-condition-asyn}
%
\begin{equation}
\eta_t \equiv \eta = \frac{c_1\log^3 T}{(1-\gamma) T\mumin} \label{eq:thm:infinite-horizaon-eta-asyn}
\end{equation}
%
for some universal constants $0<c_{1}\le 1$. 
Assume that the total number of iterations $T$ obeys
%
\begin{equation}
	T\ge\frac{c_{2} \log^2\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}}{\mumin}\max\bigg\{\frac{\log^{3}T}{(1-\gamma)^{4}\varepsilon^{2}}, \frac{t_{\mathsf{mix}}}{1-\gamma}\bigg\}
	\label{eq:thm:infinite-horizon-T-asyn}
\end{equation}
%
for some sufficiently large universal constant $c_{2}>0$. 
\end{subequations}
%
If the initialization obeys $0\leq {Q}_{0}(s,a) \leq\frac{1}{1-\gamma}$ for all $(s,a)\in \cS\times \cA$, 
then asynchronous Q-learning (cf.~\eqref{eq:async-Q-update-rule}) satisfies
%
\[
	\max_{(s,a)\in \cS\times \cA} \big| {Q}_{T}(s,a) -  {Q}^{\star}(s,a) \big| \le\varepsilon
\]
%
 with probability at least $1-\delta$. 
\end{theorem}

\begin{remark} \label{remark:expected-error-asynQ}
	Similar to Remark~\ref{remark:expected-error-inf-td} and Remark~\ref{remark:expected-error-inf}, one can immediately translate 
	the above
high-probability result into the following  mean estimation error bound: 
	%
	\begin{equation}
		\mathbb{E}\Big[ \max_{s,a} \big| {Q}_{T}(s,a) -  {Q}^{\star}(s,a) \big| \Big] \le \varepsilon (1-\delta)+\delta \frac{1}{1-\gamma}
		\leq 2\varepsilon,
	\end{equation}
	%
	which holds as long as $T\ge\frac{c_{2} \log^2\frac{|\mathcal{S}||\mathcal{A}|T}{\varepsilon(1-\gamma)}}{\mumin}\max\Big\{\frac{\log^{3}T}{(1-\gamma)^{4}\varepsilon^{2}}, \frac{t_{\mathsf{mix}}}{1-\gamma}\Big\}$ for some large enough constant $c_2>0$. 
\end{remark}

This theorem demonstrates that with high probability, the total sample size needed for asynchronous Q-learning to yield entrywise $\varepsilon$ accuracy is 
%
\begin{align}
	\widetilde{O}\bigg( \frac{1}{\mumin(1-\gamma)^{4}\varepsilon^{2}} + \frac{t_{\mathsf{mix}}}{\mumin(1-\gamma)} \bigg),
	\label{eq:sample-size-async-Q}
\end{align}
%
provided that the learning rates are taken to be some proper constant (see~\eqref{eq:thm:infinite-horizaon-eta-asyn}). 
The first term in \eqref{eq:sample-size-async-Q} resembles our sample complexity characterization of synchronous Q-learning (cf.~\eqref{eq:sample-complexity-sync-Q-learning}), except that we replace the number $|\cS||\cA|$ of state-action pairs  in \eqref{eq:sample-complexity-sync-Q-learning} with $1/\mumin$ in order to account for non-uniformity across state-action pairs.  
The second term in \eqref{eq:sample-size-async-Q} is nearly independent of the target accuracy (except for some logarithmic scaling), 
and can be viewed as the burn-in time taken for asynchronous Q-learning to mimic synchronous Q-learning despite Markovian data.   
 



We now pause to  compare Theorem~\ref{thm:infinite-horizon-asyn} with prior non-asymptotic theory for asynchronous Q-learning. As far as we know, all existing sample complexity bounds \citep{beck2012error,qu2020finite,li2020sample,even2003learning,chen2021lyapunov} scale at least as $\frac{1}{(1-\gamma)^5}$ in terms of the dependency on the effective horizon, with Theorem~\ref{thm:infinite-horizon-asyn} being the first result to sharpen this dependency to $\frac{1}{(1-\gamma)^4}$. 
In particular, our sample complexity bound strengthens the state-of-the-art result \citet{li2020sample} by a factor up to $\frac{1}{1-\gamma}$,  
while improving upon \citet{qu2020finite} by a factor of at least $\frac{|\cS||\cA|}{1-\gamma} \min\big\{\tmix, \frac{1}{(1-\gamma)^3\varepsilon^2} \big\}$.\footnote{The sample complexity of \citet{li2020sample} scales as $\widetilde{O}\big( \frac{1}{\mumin(1-\gamma)^{5}\varepsilon^{2}} + \frac{t_{\mathsf{mix}}}{\mumin(1-\gamma)} \big)$, 
while	the sample complexity of \citet{qu2020finite} scales as $\widetilde{O}\big( \frac{\tmix}{\mumin^2 (1-\gamma)^5 \varepsilon^2} \big)$. It is worth noting that $1/\mumin \geq |\cS||\cA|$ and is therefore a large factor.} 




AsynQ_analysis.tex
\section{Analysis for asynchronous Q-learning (Theorem~\ref{thm:infinite-horizon-asyn})}
\label{sec:AsynQ_analysis}




\subsection{Notation and preliminary facts}


\paragraph{Vector and matrix notation.}

We shall adopt the vector notation $\bm{Q}_t \in \mathbb{R}^{|\cS||\cA| }$, $\bm{V}_t\in \mathbb{R}^{|\cS|}$, $\bm{r}\in \mathbb{R}^{|\cS||\cA| }$ in the same way as in Section~\ref{subsec:matrix-notation}. 
The sample transition matrix $\bm{P}_t \in \mathbb{R}^{|\cS||\cA| \times |\cS|}$ in the asynchronous case is defined such that
%
\begin{equation}
	\bm{P}_t\big( (s,a), s^{\prime} \big) = \begin{cases} 1, \quad &\text{if } (s,a,s^{\prime}) = (s_{t-1},a_{t-1}, s_t); \\ 
						0, \quad & \text{else}.\end{cases}
	\label{eq:defn-Pt-async-Q}
\end{equation}
%
It is also handy to introduce the diagonal matrix 
$\bm{\Lambda}_t \in \mathbb{R}^{|\cS||\cA| \times |\cS||\cA|}$ such that
%
\begin{equation}
	\label{eq:defn-Lambda-t-async}
	\bm{\Lambda}_t\big( (s,a), (s,a) \big) = \begin{cases} \eta, \qquad & \text{if }(s, a) = (s_{t-1}, a_{t-1}); \\
							0, & \text{otherwise.}
	\end{cases}
\end{equation}
%
% the $(s, a)-$th diagonal entry $\Lambda_t(s, a) = 1$ if $(s, a) = (s_{t-1}, a_{t-1})$, and $\Lambda_t(s, a) = 0$ otherwise.
%
Armed with the above notation, the asynchronous Q-learning update rule \eqref{eq:async-Q-update-rule} can be conveniently expressed as follows:
%
\begin{align}
	\bm{Q}_{t} =\big(\bm{I}-\bm{\Lambda}_{t}\big)\bm{Q}_{t-1}+\bm{\Lambda}_{t}\big(\bm{r}+\gamma\bm{P}_{t}\bm{V}_{t-1}\big). 
	\label{eq:asynQ}
\end{align}
%


% In addition, define the transition matrix $\bm{P}_t \in \mathbb{R}^{|\cS||\cA| \times |\cS|}$ with $P_t(s^{\prime}\mymid s, a) = 1$ if $(s, a, s^{\prime}) = (s_{t-1}, a_{t-1}, s_t)$, and zeros for all other entries.


\paragraph{Range of $V_t$ and $Q_t$.}

Similar to the synchronous counterpart, we have the following elementary properties.
%
\begin{lemma}
Suppose that $0<\eta_t\leq 1$ for all $t\geq 0$. Assume that $\bm{0}\leq \bm{Q}_0\leq \frac{1}{1-\gamma} \bm{1}$. 
Then for any $t \ge 0$, one has
%
\begin{align}
	\bm{0} \le \bm{Q}_t \le \frac{1}{1-\gamma} \bm{1} \qquad \text{and} \qquad \bm{0}\leq \bm{V}_t \le \frac{1}{1-\gamma} \bm{1}.
\end{align}
%
\end{lemma}
%
\begin{proof} The proof is the same as that of Lemma~\ref{lemma:non-negativity-Qt-Vt}, and is hence omitted for brevity. \end{proof} 
	
	
\subsection{Main steps for proving Theorem~\ref{thm:infinite-horizon-asyn}}


We are now in a position to outline the main steps for the proof of Theorem~\ref{thm:infinite-horizon-asyn}. 


\paragraph{Step 1: deriving basic recursions.}


According to the update rule~\eqref{eq:asynQ}, we can derive the following elementary decomposition
%
\begin{align}
\bm{\Delta}_{t} \defn \bm{Q}_{t}-\bm{Q}^{\star} & =\big(\bm{I}-\bm{\Lambda}_{t}\big)\bm{Q}_{t-1}+\bm{\Lambda}_{t}\big(\bm{r}+\gamma\bm{P}_{t}\bm{V}_{t-1}\big)-\bm{Q}^{\star}\nonumber \\
 & =\big(\bm{I}-\bm{\Lambda}_{t}\big)\big(\bm{Q}_{t-1}-\bm{Q}^{\star}\big)+\bm{\Lambda}_{t}\big(\bm{r}+\gamma\bm{P}_{t}\bm{V}_{t-1}-\bm{Q}^{\star}\big)\nonumber \\
 & =\big(\bm{I}-\bm{\Lambda}_{t}\big)\big(\bm{Q}_{t-1}-\bm{Q}^{\star}\big)+\gamma\bm{\Lambda}_{t}\big(\bm{P}_{t}\bm{V}_{t-1}-\bm{P}\bm{V}^{\star}\big)\nonumber \\
 & =\big(\bm{I}-\bm{\Lambda}_{t}\big)\bm{\Delta}_{t-1}+\gamma\bm{\Lambda}_{t}\big(\bm{P}_{t}-\bm{P}\big)\bm{V}_{t-1}+\gamma\bm{\Lambda}_{t}\bm{P}\big(\bm{V}_{t-1}-\bm{V}^{\star}\big), \label{eq:iteration-asynQ}
\end{align}
%
where the penultimate identity follows from the Bellman optimality equation $\bm{Q}^{\star} = \bm{r} + \bm{P}\bm{V}^{\star}$.
Combining \eqref{eq:iteration-asynQ} with the inequalities \eqref{eq:V2Q-infinite} and using the definition \eqref{eq:defn-pit-s} of $\pi_t$ result in
%
\begin{subequations}
\begin{align}
\bm{\Delta}_{t} &\le \big(\bm{I}-\bm{\Lambda}_{t}\big)\bm{\Delta}_{t-1}+\gamma\bm{\Lambda}_{t}\big(\bm{P}_{t}-\bm{P}\big)\bm{V}_{t-1}+\gamma\bm{\Lambda}_{t}\bm{P}^{\pi_{t-1}}\bm{\Delta}_{t-1}, \\
\bm{\Delta}_{t} &\ge \big(\bm{I}-\bm{\Lambda}_{t}\big)\bm{\Delta}_{t-1}+\gamma\bm{\Lambda}_{t}\big(\bm{P}_{t}-\bm{P}\big)\bm{V}_{t-1}+\gamma\bm{\Lambda}_{t}\bm{P}^{\pi^{\star}}\bm{\Delta}_{t-1}.
\end{align}
\end{subequations}
%
Apply the above two relations recursively to obtain
%
\begin{subequations}
\label{eq:Delta-allstep-UB-LB-asynQ}
\begin{align}
\bm{\Delta}_{t} &\le \gamma\sum_{i=1}^{t}\prod_{j=i+1}^{t}\big(\bm{I}-\bm{\Lambda}_{j}\big)\bm{\Lambda}_{i}\big(\bm{P}_{i}-\bm{P}\big)\bm{V}_{i-1}
	+ \gamma\sum_{i=1}^{t}\prod_{j=i+1}^{t}\big(\bm{I}-\bm{\Lambda}_{j}\big)\bm{\Lambda}_{i}\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}+
	\prod_{j=1}^{t}\big(\bm{I}-\bm{\Lambda}_{j}\big)\bm{\Delta}_{0}, \label{eq:Delta-allstep-UB-asynQ}\\
\bm{\Delta}_{t} &\ge \gamma\sum_{i=1}^{t}\prod_{j=i+1}^{t}\big(\bm{I}-\bm{\Lambda}_{j}\big)\bm{\Lambda}_{i}\big(\bm{P}_{i}-\bm{P}\big)\bm{V}_{i-1}
	+ \gamma\sum_{i=1}^{t}\prod_{j=i+1}^{t}\big(\bm{I}-\bm{\Lambda}_{j}\big)\bm{\Lambda}_{i}\bm{P}^{\pi^{\star}}\bm{\Delta}_{i-1}+
	\prod_{j=1}^{t}\big(\bm{I}-\bm{\Lambda}_{j}\big)\bm{\Delta}_{0}.
	\label{eq:Delta-allstep-LB-asynQ}
\end{align}
\end{subequations}
%


By defining the following diagonal matrices
%
\begin{equation}
\bm \Lambda_{i}^{(t)}\coloneqq\begin{cases}
\prod_{j=1}^{t}(\bm I-\bm \Lambda_{j}), & \text{if }i=0 ,\\
\bm \Lambda_{i}\prod_{j=i+1}^{t}(\bm I-\bm \Lambda_{j}), & \text{if }0<i<t ,\\
\bm \Lambda_{t}, & \text{if }i=t ,
\end{cases}\label{def:Lambda-i-t}
\end{equation}
%
and setting 
%
\begin{equation}
	\beta = \frac{c_3(1-\gamma)}{\log T}
	\label{eq:defn-beta-async-Q}
\end{equation}
%
for some constant $c_3 > 0$, 
we can rearrange terms in the upper bound \eqref{eq:Delta-allstep-UB-asynQ} to reach
\begin{align}
	\bm{\Delta}_{t} &\le
	\underset{\eqqcolon\,\bm{\zeta}_{t}}{\underbrace{\bm \Lambda_{0}^{(t)}\bm{\Delta}_{0} + \gamma\sum_{i=1}^{(1-\beta)t}\bm \Lambda_{i}^{(t)}\Big[\big(\bm{P}_{i}-\bm{P}\big)\bm{V}_{i-1} + \bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}\Big]}} \nonumber\\
	&\qquad+ \underset{\eqqcolon\,\bm{\xi}_{t}}{\underbrace{\gamma\sum_{i=(1-\beta)t+1}^{t}\bm \Lambda_{i}^{(t)}\big(\bm{P}_{i}-\bm{P}\big)\bm{V}_{i-1}}} + \gamma\sum_{i=(1-\beta)t+1}^{t}\bm \Lambda_{i}^{(t)}\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}. 
	\label{eq:defn-main}
\end{align}
%
%Similar to the proof of Lemma~\ref{lem:inf-upper-bound}, 
In the subsequent steps, we shall first develop bounds on the sizes of the terms $\bm{\zeta}_{t}$ and $\bm{\xi}_{t}$ in \eqref{eq:defn-main} separately, and then combine these bounds with \eqref{eq:defn-main} recursively in order to derive the advertised upper bound on $\bm{\Delta}_{t}$.  



\paragraph{Step 2: bounding the terms $\bm{\zeta}_{t}$ and $\bm{\xi}_{t}$.}

The terms $\bm{\zeta}_{t}$ and $\bm{\xi}_{t}$ defined in \eqref{eq:defn-main} can be bounded with high probability by the following lemmas.

\begin{lemma} \label{lem:zeta-async} 
With probability at least $1-\delta$, we have
%
\begin{align}
\|\bm{\zeta}_{t}\|_{\infty} \leq \frac{4}{(1-\gamma)T}
\end{align}
%
for all  $t$ obeying $\frac{T}{c_{4}\log T}\leq t\leq T$. Here, $c_4>0$ is some constant obeying $c_4 \le c_1c_3/4$, 
where the constants $c_1$ and $c_3$ appear in \eqref{eq:thm:infinite-horizaon-eta-asyn} and \eqref{eq:defn-beta-async-Q}, respectively. 
\end{lemma}
%
\begin{proof} See Section~\ref{sec:proof-lem:zeta-async}. \end{proof}
%
\begin{lemma} \label{lem:xi-async}
Suppose that $0<\eta \le \frac{\log^{3}T}{(1-\gamma)T\mumin}$. With probability at least $1-\delta$, one has
\begin{align}
	|\bm{\xi}_{t}| \leq \sqrt{\frac{16\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)T\mumin}\Big(\max_{(1-\beta)t\leq i<t}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})+\bm{1}\Big)}+\frac{6\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{2}T\mumin}\bm{1}
\end{align}
%
for all  $t$ obeying $\frac{T}{c_{4}\log T}\leq t\leq T$ for some constant $c_4>0$.  
\end{lemma}
%
\begin{proof} See Section~\ref{sec:proof-lem:xi-async}. \end{proof}


\paragraph{Step 3: controlling $\bm{\Delta}_{t}$.}
%
Consider any $t$ obeying $\frac{T}{c_{4}\log T}\leq t\leq T$ and any $k$ obeying $2t/3 \leq k\leq t$.   
Under the sample size condition \eqref{eq:thm:infinite-horizon-T-asyn},  Lemmas~\ref{lem:zeta-async}-\ref{lem:xi-async} together with a little algebra lead to
%
\[
	|\bm{\zeta}_{k}| + |\bm{\xi}_{k}|  \leq 
\sqrt{\frac{32\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)T\mumin}\bigg(\max_{(1-\beta)k\le i\leq k}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})+\bm{1}\bigg)} 
	\leq \sqrt{\bm{\varphi}_{t}}, 
\]
%
where we define
%
\begin{equation}
	\bm{\varphi}_{t}\coloneqq \frac{ 32 \big(\log^{3}T \big)\big( \log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta} \big) }{(1-\gamma)T\mumin}\bigg(\max_{\frac{t}{2}\le i\leq t}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})+\bm{1}\bigg). \label{def:phi-asynQ}
\end{equation}
%
Combining this inequality with \eqref{eq:defn-main} allows us to obtain
%
\begin{align}
\bm{\Delta}_{k} & \le\sqrt{\bm{\varphi}_{t}}+\sum_{i=(1-\beta)k+1}^{k}\bm \Lambda_{i}^{(k)}\gamma\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}=\sqrt{\bm{\varphi}_{t}}+\sum_{i=(1-\beta)k}^{k-1}\bm \Lambda_{i+1}^{(k)}\gamma\bm{P}^{\pi_{i}}\bm{\Delta}_{i}\qquad\text{for all }~2t/3\le k\le t.\label{eq:Deltak-UB-phit-infinite-asynQ}
\end{align}
%


Similar to the quantity $\alpha_{i}^{(t)}$ defined in \eqref{eq:defn-alpha-it-sync}, let us define
%
\begin{align}
	\bm D_{i}^{(t)} \defn \bigg( \sum_{j=(1-\beta)t}^{t-1}\bm \Lambda_{j+1}^{(t)}\bigg)^{-1}\bm \Lambda_{i+1}^{(t)} ,
	\label{eq:defn-Dit-async-Q}
\end{align}
%
which, according to \eqref{eq:sum-eta-i-t-infinite} and the definition \eqref{eq:defn-Lambda-t-async}, clearly satisfies
%
\begin{equation}
	\bm D_{i}^{(t)} \geq \bm{\Lambda}_{i+1}^{(t)} \geq \bm{0} \qquad \text{and} \qquad 
	\sum_{i=(1-\beta)t}^{t-1}\bm D_{i}^{(t)} = \bm{I}. 
	\label{eq:Dit-property-async}
\end{equation}
%
Set $i_0=t$ for notational convenience. With this set of notation and the property \eqref{eq:Dit-property-async} in mind, we can derive the following bound
%
\begin{align}
\bm{\Delta}_{t} & \le\sum_{i_{1}=(1-\beta)t}^{t-1}\Big(\bm D_{i_{1}}^{(t)}\sqrt{\bm{\varphi}_{t}}+\bm \Lambda_{i_{1}+1}^{(t)}\gamma\bm{P}^{\pi_{i_{1}}}\bm{\Delta}_{i_{1}}\Big)\nonumber \\
 & \leq\sum_{i_{1}=(1-\beta)t}^{t-1}\Bigg[\bm D_{i_{1}}^{(t)}\sqrt{\bm{\varphi}_{t}}+\bm \Lambda_{i_{1}+1}^{(t)}\gamma\bm{P}^{\pi_{i_{1}}}\sum_{i_{2}=(1-\beta)i_{1}}^{i_{1}-1}\Big(\bm D_{i_{2}}^{(i_{1})}\sqrt{\bm{\varphi}_{t}}+\bm \Lambda_{i_{2}+1}^{(i_{1})}\gamma\bm{P}^{\pi_{i_{2}}}\bm{\Delta}_{i_{2}}\Big)\Bigg]\nonumber \\
 & \leq\sum_{i_{1}=(1-\beta)t}^{t-1}\bm D_{i_{1}}^{(t)}\sqrt{\bm{\varphi}_{t}}+\sum_{i_{1}=(1-\beta)t}^{t-1}\bm D_{i_{1}}^{(t)}\sum_{i_{2}=(1-\beta)i_{1}}^{i_{1}-1}\big(\gamma\bm{P}^{\pi_{i_{1}}}\bm D_{i_{2}}^{(i_{1})}\big)\sqrt{\bm{\varphi}_{t}} \nonumber \\
 & \qquad\qquad +\sum_{i_{1}=(1-\beta)t}^{t-1}\sum_{i_{2}=(1-\beta)i_{1}}^{i_{1}-1}\prod_{k=1}^{2}\big(\bm \Lambda_{i_{k}+1}^{(i_{k-1})}\gamma\bm{P}^{\pi_{i_{k}}}\big)\bm{\Delta}_{i_{2}}\nonumber \\
 & =\sum_{i_{1}=(1-\beta)t}^{t-1}\bm D_{i_{1}}^{(t)}\Bigg\{\bm I + \sum_{i_{2}=(1-\beta)i_{1}}^{i_{1}-1} \gamma\bm{P}^{\pi_{i_{1}}}\bm D_{i_{2}}^{(i_{1})} \Bigg\}\sqrt{\bm{\varphi}_{t}}+\sum_{i_{1}=(1-\beta)t}^{t-1}\sum_{i_{2}=(1-\beta)i_{1}}^{i_{1}-1}\prod_{k=1}^{2}\big(\bm \Lambda_{i_{k}+1}^{(i_{k-1})}\gamma\bm{P}^{\pi_{i_{k}}}\big)\bm{\Delta}_{i_{2}}.\label{eq:Delta-recursion-inf-asynQ}
\end{align}
%
Here, the first relation makes use of the second property in \eqref{eq:Dit-property-async}, 
the second relation further expands $\bm{\Delta}_{i_{1}}$ in the same way as in the first line of \eqref{eq:Delta-recursion-inf-asynQ}, 
whereas the third inequality relies on the first property in \eqref{eq:Dit-property-async}.  


Next, we intend to invoke the above relation multiple times to reach a simpler relation. 
Set
%
\begin{align}
	H \defn \frac{\log T}{1-\gamma}. 
\end{align}
%
Similar to the way we derive \eqref{eq:Delta-UB-temp-inf}, we can apply the relation \eqref{eq:Delta-recursion-inf-asynQ} recursively and use the basic relation $|\bm{\Delta}_{k}|\leq \frac{1}{1-\gamma}\bm{1}$ for any $k$ to show that
%
\begin{align}
\bm{\Delta}_{t} & \le\sum_{\left(i_{1},\cdots,i_{H}\right)\in\mathcal{I}_{t}}\Bigg\{\bm{D}_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-1}\gamma^{h}\prod_{k=1}^{h}\big(\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)\bigg)\sqrt{\bm{\varphi}_{t}}+\gamma^{H}\prod_{k=1}^{H}\big(\bm{\Lambda}_{i_{k}+1}^{(i_{k-1})}\bm{P}^{\pi_{i_{k}}}\big)\big|\bm{\Delta}_{i_{H}}\big|\Bigg\},
	\label{eq:Delta-UB-temp-inf-asynQ-1}
\end{align}
%
where $\mathcal{I}_{t}$ has been defined in~\eqref{eq:defn-It-inf}. To further simplify \eqref{eq:Delta-UB-temp-inf-asynQ-1}, 
we need to control the two terms on the right-hand side of \eqref{eq:Delta-UB-temp-inf-asynQ-1} separately. 
%
\begin{itemize}

	\item We shall begin with the first term on the right-hand side of \eqref{eq:Delta-UB-temp-inf-asynQ-1}. Towards this end, let us define a collection of policies $\{\widehat{\pi}_k\}$ recursively and backward as follows: 
%
\begin{align}
\widehat{\pi}_k \defn
	\begin{cases} \arg\max_{\pi \in \Pi} \bm{\Pi}^{\pi}\sqrt{\bm{\varphi}_{t}}, & \text{if }k=H-1, \\
		\arg\max_{\pi \in \Pi} \bm \Pi^{\pi}\Big(\bm{I}+\sum_{h=k+1}^{H-1}\gamma^{h}\prod_{j=1}^{h}\bm{P}^{\widehat{\pi}_j}\Big)\sqrt{\bm{\varphi}_{t}} , \quad & \text{if }k= H-2, \cdots, 1,
	\end{cases}
	\label{eq:defn-hat-pi-Pi}
\end{align}
%
or alternatively (in view of the definition \eqref{eqn:ppivq} of $\bm{P}^{\pi}$), 		
%
\begin{align}
\widehat{\pi}_k \defn
	\begin{cases} \arg\max_{\pi \in \Pi} \bm P^{\pi}\sqrt{\bm{\varphi}_{t}}, & \text{if }k=H-1; \\
		\arg\max_{\pi \in \Pi} \bm P^{\pi}\Big(\bm{I}+\sum_{h=k+1}^{H-1}\gamma^{h}\prod_{j=1}^{h}\bm{P}^{\widehat{\pi}_j}\Big)\sqrt{\bm{\varphi}_{t}} , \quad & \text{if }k= H-2, \cdots, 1. 
	\end{cases}
	\label{eq:defn-hat-pi-P}
\end{align}
%
Here, $\Pi$ is a policy set satisfying
%
\begin{eqnarray}
\label{defn:Pi-asyncQ}
\begin{aligned}
	\Pi  \defn\big\{\pi=[\pi(s)]_{s\in\cS}\, \big| \, \pi(s) \in\Pi_{s}, \forall s\in\cS\big\}, 
	\qquad \Pi_{s}  \defn\big\{ \pi_{i}(s) \mid i\in[t/2,t)\big\};	 
\end{aligned}
\end{eqnarray}
%
in words, for any policy $\pi$ belonging to $\Pi$, each $\pi(s)$ coincides with one of the policy iterates $\pi_i(s)$ during the latest $\beta t$ iterations, although we do not require all $\{\pi(s)\}$ across different states to be associated with the same time stamp $i$.  
% Note that $\pi$ can be optimized separately for each state, and hence the definition \eqref{eq:defn-hat-pi-Pi} makes sense (so that $\arg\max$ can be computed for each state separately). 
%
With this collection of policies in place, we can deduce that
%
\begin{align*}
 & \sum_{i_{1},\cdots,i_{H}}\bm{D}_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-1}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)\bigg)\sqrt{\bm{\varphi}_{t}}\\
 & =\sum_{i_{1},\cdots,i_{H-1}}\sum_{i_{H}}\bm{D}_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-2}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)+\prod_{k=1}^{H-2}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)\Big(\gamma\bm{P}^{\pi_{i_{H-1}}}\bm{D}_{i_{H}}^{(i_{H-1})}\Big)\bigg)\sqrt{\bm{\varphi}_{t}}\\
 & =\sum_{i_{1},\cdots,i_{H-1}}\bm{D}_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-2}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)+\prod_{k=1}^{H-2}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)\Big(\gamma\bm{P}^{\pi_{i_{H-1}}}\sum_{i_{H}}\bm{D}_{i_{H}}^{(i_{H-1})}\Big)\bigg)\sqrt{\bm{\varphi}_{t}}\\
 & \overset{(\mathrm{i})}{=}\sum_{i_{1},\cdots,i_{H-1}}\bm{D}_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-2}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)+\prod_{k=1}^{H-2}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)\gamma\bm{P}^{\pi_{i_{H-1}}}\bigg)\sqrt{\bm{\varphi}_{t}}\\
 & \overset{(\mathrm{ii})}{\leq}\sum_{i_{1},\cdots,i_{H-1}}\bm{D}_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-2}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)+\prod_{k=1}^{H-2}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)\gamma\bm{P}^{\widehat{\pi}_{H-1}}\bigg)\sqrt{\bm{\varphi}_{t}}\\
 %
 & =\sum_{i_{1},\cdots,i_{H-2}}\bm{D}_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-3}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)+\prod_{k=1}^{H-3}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)\gamma\bm{P}^{\pi_{i_{H-2}}}\bigg(\sum_{i_{H-1}}\bm{D}_{i_{H-1}}^{(i_{H-2})}\bigg)\big(\bm{I}+\gamma\bm{P}^{\widehat{\pi}_{H-1}}\big)\bigg)\sqrt{\bm{\varphi}_{t}}\\
 %
 & \overset{(\mathrm{iii})}{=}\sum_{i_{1},\cdots,i_{H-2}}\bm{D}_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-3}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)+\prod_{k=1}^{H-3}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)\gamma\bm{P}^{\pi_{i_{H-2}}}\big(\bm{I}+\gamma\bm{P}^{\widehat{\pi}_{H-1}}\big)\bigg)\sqrt{\bm{\varphi}_{t}}\\
 %
 & \overset{(\mathrm{iv})}{\leq}\sum_{i_{1},\cdots,i_{H-2}}\bm{D}_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-3}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)+\prod_{k=1}^{H-3}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)\gamma\bm{P}^{\widehat{\pi}_{H-2}}\big(\bm{I}+\gamma\bm{P}^{\widehat{\pi}_{H-1}}\big)\bigg)\sqrt{\bm{\varphi}_{t}}, 
\end{align*}
%
where we abbreviate $\sum_{(i_{1},\cdots,i_{H})\in \mathcal{I}_t}$ as $\sum_{i_{1},\cdots,i_{H}}$ 
as long as it is clear from the context.  Here, (i) and (iii) arise from the second property in \eqref{eq:Dit-property-async}, 
while (ii) and (iv) are due to the construction \eqref{eq:defn-hat-pi-P}. 
Continuing the derivation of the above inequality recursively, we arrive at  
%
\begin{align*}
\sum_{i_{1},\cdots,i_{H}}\bm{D}_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-1}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)\bigg)\sqrt{\bm{\varphi}_{t}} & \le\bigg(\bm{I}+\sum_{h=1}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\widehat{\pi}_{k}}\bigg)\sqrt{\bm{\varphi}_{t}}.
\end{align*}



%\begin{align*}
%&\sum_{\left(i_{1},\cdots,i_{H}\right)\in\mathcal{I}_{t}}\bm D_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-1}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm D_{i_{k+1}}^{(i_{k})}\big)\bigg)\sqrt{\bm{\varphi}_{t}} \\
%&\qquad= \sum_{\left(i_{1},\cdots,i_{H-1}\right)\in\mathcal{I}_{t}}\bm D_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-2}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm D_{i_{k+1}}^{(i_{k})}\big) + \prod_{k=1}^{H-2}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm D_{i_{k+1}}^{(i_{k})}\big)\gamma\bm{P}^{\pi_{i_{H-1}}}\bigg)\sqrt{\bm{\varphi}_{t}} \\
%&\qquad\le \sum_{\left(i_{1},\cdots,i_{H-1}\right)\in\mathcal{I}_{t}}\bm D_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-2}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm D_{i_{k+1}}^{(i_{k})}\big) + \prod_{k=1}^{H-2}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm D_{i_{k+1}}^{(i_{k})}\big)\gamma\bm{P}^{\widehat{\pi}_{H-1}}\bigg)\sqrt{\bm{\varphi}_{t}} \\
%&\qquad= \sum_{\left(i_{1},\cdots,i_{H-2}\right)\in\mathcal{I}_{t}}\bm D_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-3}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm D_{i_{k+1}}^{(i_{k})}\big) + \prod_{k=1}^{H-3}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm D_{i_{k+1}}^{(i_{k})}\big)\gamma\bm{P}^{\pi_{i_{H-2}}}\big(\bm I + \gamma\bm{P}^{\widehat{\pi}_{H-1}}\big)\bigg)\sqrt{\bm{\varphi}_{t}} \\
%&\qquad\le \sum_{\left(i_{1},\cdots,i_{H-2}\right)\in\mathcal{I}_{t}}\bm D_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-3}\prod_{k=1}^{h}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm D_{i_{k+1}}^{(i_{k})}\big) + \prod_{k=1}^{H-3}\big(\gamma\bm{P}^{\pi_{i_{k}}}\bm D_{i_{k+1}}^{(i_{k})}\big)\gamma\bm{P}^{\widehat{\pi}_{H-2}}\big(\bm I + \gamma\bm{P}^{\widehat{\pi}_{H-1}}\big)\bigg)\sqrt{\bm{\varphi}_{t}} \\
%&\qquad\le \ldots \\
%&\qquad\le \bigg(\bm{I}+\sum_{h=1}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\widehat{\pi}_k}\bigg)\sqrt{\bm{\varphi}_{t}}. 
%\end{align*}
%%

\item

We now turn attention to the second term on the right-hand side of \eqref{eq:Delta-UB-temp-inf-asynQ-1}. It is seen that
%
\begin{align*}
\sum_{i_{1},\cdots,i_{H}}\prod_{k=1}^{H} \gamma^H \big(\bm{\Lambda}_{i_{k}+1}^{(i_{k-1})}\bm{P}^{\pi_{i_{k}}}\big)\big|\bm{\Delta}_{i_{H}}\big| & \leq\frac{\gamma^H}{1-\gamma}\sum_{i_{1},\cdots,i_{H}}\prod_{k=1}^{H}\big(\bm{D}_{i_{k}}^{(i_{k-1})}\bm{P}^{\pi_{i_{k}}}\big)\bm{1}\\
& = \frac{\gamma^H}{1-\gamma}\sum_{i_{1},\cdots,i_{H-1}}\prod_{k=1}^{H-1}\big(\bm{D}_{i_{k}}^{(i_{k-1})}\bm{P}^{\pi_{i_{k}}}\big)\bigg(\sum_{i_{H}}\bm{D}_{i_{H}}^{(i_{H-1})} \big( \bm{P}^{\pi_{i_{H}}}\bm{1} \big)\bigg)\\
%
 & = \frac{\gamma^H}{1-\gamma}\sum_{i_{1},\cdots,i_{H-1}}\prod_{k=1}^{H-1}\big(\bm{D}_{i_{k}}^{(i_{k-1})}\bm{P}^{\pi_{i_{k}}}\big)\bigg(\sum_{i_{H}}\bm{D}_{i_{H}}^{(i_{H-1})}\bigg) \bm{1} \\
 & =\frac{\gamma^H}{1-\gamma}\sum_{i_{1},\cdots,i_{H-1}}\prod_{k=1}^{H-1}\big(\bm{D}_{i_{k}}^{(i_{k-1})}\bm{P}^{\pi_{i_{k}}}\big)\bm{1}\\
	& =\cdots =\frac{\gamma^H}{1-\gamma}\bm{1} = \frac{\gamma^{\frac{\log T}{1-\gamma}}}{1-\gamma}\bm{1} \\
	& \leq \frac{1}{(1-\gamma)T} \bm{1},
\end{align*}
%
where the first line follows from the first property in \eqref{eq:Dit-property-async}, 
the third line is due to the fact $\bm{P}^{\pi}\bm{1}=\bm{1}$ for any $\pi$,  
and the fourth line arises from the second property in \eqref{eq:Dit-property-async}. 


\end{itemize}
%
Substituting the above two bounds into \eqref{eq:Delta-UB-temp-inf-asynQ-1} yields
%
\begin{align}
\bm{\Delta}_{t} 
%& \le\sum_{\left(i_{1},\cdots,i_{H}\right)\in\mathcal{I}_{t}}\Bigg\{\bm{D}_{i_{1}}^{(t)}\bigg(\bm{I}+\sum_{h=1}^{H-1}\gamma^{h}\prod_{k=1}^{h}\big(\bm{P}^{\pi_{i_{k}}}\bm{D}_{i_{k+1}}^{(i_{k})}\big)\bigg)\sqrt{\bm{\varphi}_{t}}+\gamma^{H}\prod_{k=1}^{H}\big(\bm{\Lambda}_{i_{k}+1}^{(i_{k-1})}\bm{P}^{\pi_{i_{k}}}\big)\big|\bm{\Delta}_{i_{H}}\big|\Bigg\}\nonumber\\
	& \leq\underbrace{\bigg(\bm{I}+\sum_{h=1}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\widehat{\pi}_{k}}\bigg)\sqrt{\bm{\varphi}_{t}}}_{\eqqcolon \,\bm{\beta}}+\frac{1}{(1-\gamma)T}\bm{1}. 
	\label{eq:Delta-UB-temp-inf-asynQ}
\end{align}
%




\paragraph{Step 4: putting all pieces together.}
Repeating our analysis for the term $\bm{\beta}_1$ in Section~\ref{sec:proof_inf_upper_bound} (i.e., Step 5 of Section~\ref{sec:proof_inf_upper_bound} with Lemma~\ref{lemma:inf-alpha-claim} replaced by Lemma~\ref{lemma:inf-alpha-claim-async}), 
we arrive at 
%
\begin{align*}
|\bm{\beta}|^{2} 
& \leq 
\frac{320\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2 (1-\gamma)^{4}T}\Big(1+2\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\Big)\bm{1} 
\end{align*}
%
with probability at least $1-\delta$. 
Substitution into \eqref{eq:Delta-UB-temp-inf-asynQ} then yields
%
\begin{align}
\notag \bm{\Delta}_{t} &\leq 
 \sqrt{\frac{320\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2 (1-\gamma)^{4}T\mumin}\Big(1+\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\Big)}\ \bm{1} + \frac{1}{(1-\gamma)T}\bm{1} \\ 
 &\leq 30 \sqrt{\frac{\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2 (1-\gamma)^{4}T\mumin}\Big(1+\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\Big)}\ \bm{1}
	\label{eq:Delta-UB-final-asynQ}
\end{align}
%
holds simultaneously for all $t\ge\frac{T}{c_{4}\log T}$, provided that the sample size condition \eqref{eq:thm:infinite-horizon-T-asyn} is satisfied. 
Similarly, we can also establish the following lower bound on $\bm{\Delta}_{t}$ (which we omit the details for the sake of brevity)
%
\begin{align*}
\bm{\Delta}_{t} \geq -30 \sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2 (1-\gamma)^{4}T\mumin}\Big(1+\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\Big)}\ \bm{1} 
	% \label{eq:Delta-LB-final-asynQ}
\end{align*}
%
with probability at least $1-\delta$. To summarize, it is seen that with probability exceeding $1-2\delta$, 
%
\begin{align}
	\|\bm{\Delta}_{t} \|_{\infty} \leq 30 \sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2 (1-\gamma)^{4}T\mumin}\Big(1+\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\Big)}. 
	\label{eq:Delta-UB-LB-final-asynQ}
\end{align}
%
This resembles the relation \eqref{eq:Delta-t-inf-recursion-inf-outline} derived for the synchronous case, 
except that $T$ in the denominator is replaced with $\mumin T$. 
As a result, we can readily repeat the argument in Appendix~\ref{sec:combine_ub_lb} to reach
%
\begin{align}
	\|\bm{\Delta}_{T}\|_{\infty} \leq O\bigg( \sqrt{\frac{\big(\log^{4}T  \big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{4}T\mumin}} +\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{4}T\mumin} 
	\bigg), \label{eq:Delta-T-UB-final-inf-async}
\end{align}
%
which in turn establishes the claimed result in Theorem~\ref{thm:infinite-horizon-asyn}. 

%Then, Theorem~\ref{thm:infinite-horizon-asyn} is handy by applying the arguments in Section~\ref{sec:combine_ub_lb}.


\subsection{Proofs of technical lemmas}


\subsubsection{Proof of Lemma~\ref{lem:zeta-async}}
\label{sec:proof-lem:zeta-async}

In view of the definition of $\bm{\zeta}_{t}$ in \eqref{eq:defn-main} and the fact that $\bm \Lambda_{0}^{(t)}$ is a diagonal matrix, 
we can deduce that
%
\begin{align*}
\|\bm{\zeta}_{t}\|_{\infty} 
& \leq \|\bm \Lambda_{0}^{(t)}\|\|\bm{\Delta}_{0}\|_{\infty}+t\max_{1\leq i\leq(1-\beta)t}\|\bm \Lambda_{i}^{(t)}\|\max_{1\leq i\leq(1-\beta)t}\big(\|\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}\|_{\infty}+\|\bm{P}_{i}\bm{V}_{i-1}\|_{\infty}+\|\bm{P}\bm{V}_{i-1}\|_{\infty}\big)\\
 & \leq\|\bm \Lambda_{0}^{(t)}\|\|\bm{\Delta}_{0}\|_{\infty}+t\max_{1\leq i\leq(1-\beta)t}\|\bm \Lambda_{i}^{(t)}\|\max_{1\leq i\leq(1-\beta)t}\Big\{\|\bm{P}^{\pi_{i-1}}\|_{1}\|\bm{\Delta}_{i-1}\|_{\infty}+\big(\|\bm{P}_{i}\|_{1}+\|\bm{P}\|_{1}\big)\|\bm{V}_{i-1}\|_{\infty}\Big\}\\
 & \overset{(\mathrm{i})}{=}\|\bm \Lambda_{0}^{(t)}\|\|\bm{\Delta}_{0}\|_{\infty}+t\max_{1\leq i\leq(1-\beta)t}\|\bm \Lambda_{i}^{(t)}\|\max_{1\leq i\leq(1-\beta)t}\big(\|\bm{\Delta}_{i-1}\|_{\infty}+2\,\|\bm{V}_{i-1}\|_{\infty}\big)\\
 & \overset{(\mathrm{ii})}{\leq}\frac{1}{T^{2}}\cdot\frac{1}{1-\gamma}+\frac{1}{T^{2}}\cdot t\cdot\frac{3}{1-\gamma}\\
 & \le\frac{4}{(1-\gamma)T}.
\end{align*}
%
Here, (i) holds true since $\|\bm{P}^{\pi_{i-1}}\|_{1}=\|\bm{P}_{i}\|_{1}=\|\bm{P}\|_{1}=1$. To verify (ii), we first define
%
\begin{equation}\label{eq:defn-tk-sa}
	t_{k}(s,a) \defn \text{the time stamp when }\text{the trajectory visits }(s,a)\text{ for the }k\text{-th time}
	%\text{-th time after }(1-\beta)t
\end{equation}
%
and
%
\begin{equation}
	K_{t}(s,a) \defn  \Big| \big\{ k \geq 1\mid t_{k}(s,a) < t\big\} \Big|,  \label{eq:defn-Kt}
\end{equation}
%
namely, the total number of times --- before the $t$-th iteration ---
that the sample trajectory visits $(s,a)$.
Then \citet[Lemma 8]{li2020sample} tells us that with probability at least $1-\delta$, 
%
\begin{align}
K_{t_1}(s,a)-K_{t_2}(s, a) \ge \frac{1}{2}(t_1 - t_2)\mumin,
\end{align}
%
holds uniformly for all $(s, a) \in \cS\times\cA$ and $0 \le t_2 \le t_1 \le T$ obeying 
%
$$
	t_1 - t_2 \ge \frac{886t_{\mathsf{mix}}}{\mumin}\log\frac{|\cS||\cA|T}{\delta}.
$$
%
This in turn implies that: 
if $\beta t \ge \frac{886t_{\mathsf{mix}}}{\mumin}\log\frac{|\cS||\cA|T}{\delta}$ and $i\leq (1-\beta)t$, then one has
%
\begin{subequations}
	\label{eq:Lambda-0t-Lambda-it-async-Q}
\begin{align}
	\|\bm \Lambda_{0}^{(t)}\| &= \bigg\| \prod_{j=1}^{t}\big(\bm{I}-\bm{\Lambda}_{j}\big)\bigg\| 
	= \max_{(s, a)\in \cS\times \cA}(1-\eta)^{K_{t}(s,a)} \leq (1-\eta)^{\frac{1}{2}t\mumin} \le \frac{1}{T^{2}} \\
\|\bm \Lambda_{i}^{(t)}\| &= \bigg \| \bm \Lambda_{i}\prod_{j=i+1}^{t}(\bm I-\bm \Lambda_{j})\bigg\|  \leq \max_{(s, a)} (1-\eta)^{K_{t}(s,a)-K_{i}(s, a)} \leq (1-\eta)^{\frac{1}{2}\beta t\mumin} \le \frac{1}{T^{2}}
\end{align}
\end{subequations}
%
with probability at least $1-\delta$, 
provided that $\eta\beta t\mumin > 4\log T$. 
In other words, \eqref{eq:Lambda-0t-Lambda-it-async-Q} holds with probability at least $1-\delta$, as long as
%
\[
t>\max\left\{ \frac{4\log T}{\eta\beta\mumin}, \, \frac{886\tmix}{\beta\mumin}\log\frac{|\cS||\cA|T}{\delta}\right\} =\max\left\{ \frac{4T}{c_{1}c_{3}\log T}, \, \frac{886\tmix}{c_{3}(1-\gamma)\mumin}\log\frac{|\cS||\cA|T}{\delta}\log T\right\} .  	
\]
%
This taken together with the sample size assumption \eqref{eq:thm:infinite-horizon-T-asyn} concludes the proof of Lemma~\ref{lem:zeta-async}. 



\subsubsection{Proof of Lemma~\ref{lem:xi-async}}
\label{sec:proof-lem:xi-async}

Fix any state-action pair $(s,a)\in\mathcal{S}\times\mathcal{A}$, and let us look at the $(s,a)$-th entry of $\bm{\xi}_{t}$, i.e., 
$\xi_{t}(s,a)$.
For notational simplicity, let $\bm{\Lambda}_{j}(s,a)$ denote the $(s,a)$-th diagonal entry
of the diagonal matrix $\bm{\Lambda}_{j}$, and $\bm{P}_{t}(s,a)$
(resp.~$\bm{P}(s,a)$) the $(s,a)$-th row of $\bm{P}_{t}$ (resp.~$\bm{P}$). 
 


Using the definition of $\bm{\xi}_{t}$ in \eqref{eq:defn-main} and the above notation, we can derive
%
\begin{align}
	\xi_{t}(s,a) & = \gamma \sum_{i=(1-\beta)t+1}^{t}\prod_{j=i+1}^{t}\big(1-\bm{\Lambda}_{j}(s,a)\big)\bm{\Lambda}_{i}(s,a)\big(\bm{P}_{i}(s,a)-\bm{P}(s,a)\big)\bm{V}_{i-1}.
	\label{eq:expression-xi-sa}
\end{align}
%
Equipped with the definitions of $t_k(s,a)$ (cf.~\eqref{eq:defn-tk-sa}) and $K_t(s,a)$ (cf.~\eqref{eq:defn-Kt}), 
we can further rewrite \eqref{eq:expression-xi-sa} as
%
\begin{align}
	\xi_{t}(s,a) & = \gamma \sum_{k=K_{(1-\beta)t+1}}^{K_{t}(s,a)}(1-\eta)^{K_{t}(s,a)-k}\eta\big(\bm{P}_{t_{k}+1}(s,a)-\bm{P}(s,a)\big)\bm{V}_{t_{k}}.\label{eq:expression-xi-sa-1}
\end{align}
%
In what follows, we shall suppress the notation and write $t_k = t_k(s,a)$ 
and $K_t = K_t(s,a)$  to streamline notation. 


% where we suppress the dependency on $(s,a)$ and write $t_k:=t_k(s,a)$
%$K_t: = K_t(s,a)$ 
 
The main step thus boils down to controlling \eqref{eq:expression-xi-sa-1}. 
Towards this, we claim that: with probability at least $1-\delta$,
%
\begin{align}
	& \Bigg|\sum_{k=K_{\beta}}^{K}(1-\eta)^{K-k}\eta\big(\bm{P}_{t_{k}+1}(s,a)-\bm{P}(s,a)\big)\bm{V}_{t_{k}}\Bigg| \nonumber\\
	&\qquad \leq \sqrt{\frac{16\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)T\mumin}\Big(\max_{t_{K_{\beta}}\leq i \leq t_K}\mathsf{Var}_{\bm{P}(s, a)}(\bm{V}_{i})+1\Big)}+\frac{6\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{2}T\mumin} 
	\label{eq:claim-Bernstein-fixed-K}
\end{align}
%
holds simultaneously for all $(s,a)\in\mathcal{S}\times\mathcal{A}$ and
all $1 \le K_{\beta} \le K \le T$, provided that $0<\eta \le \frac{\log^{3}T}{(1-\gamma)T\mumin}$. 
If this claim were true, then 
taking $K_{\beta} = K_{ (1-\beta)t + 1 }$ and $K = K_t$ and substituting the bound \eqref{eq:claim-Bernstein-fixed-K} into the expression \eqref{eq:expression-xi-sa-1} would lead to
%
\begin{align}
	|\bm{\xi}_{t}| \leq \sqrt{\frac{16\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)T\mumin}\Big(\max_{(1-\beta)t\leq i<t}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})+\bm{1}\Big)}+\frac{6\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{2}T\mumin}\bm{1}, 
\end{align}
%
thus concluding the proof of this lemma. To finish up, it is sufficient to justify the claim (\ref{eq:claim-Bernstein-fixed-K}), which forms the content of the remainder of this proof. 




\begin{proof}[Proof of the claim (\ref{eq:claim-Bernstein-fixed-K})]
%
Let us use the notation in \eqref{def:eta-i-t} to express
$\eta_k^{(K)} = (1-\eta)^{K-k}\eta$.  For any fixed integer $K>0$, the following vectors
\[
\left\{ \bm{P}_{t_{k}+1}(s,a)\mid1\leq k\leq K\right\} 
\]
are identically and independently distributed; see \citet[Section~B.1]{li2020sample}.
We can then express the term
\[
X_K \defn \sum_{k=K_{\beta}}^{K}(1-\eta)^{K-k}\eta\big(\bm{P}_{t_{k}+1}(s,a)-\bm{P}(s,a)\big)\bm{V}_{t_{k}},
\]
as follows: 
\[
X_K=\sum_{k=K_{\beta}}^{K}z_{k}\qquad\text{with }z_{k}\coloneqq\eta_k^{(K)}\big(\bm{P}_{t_{k}+1}(s,a)-\bm{P}(s,a)\big)\bm{V}_{t_{k}},
\]
%
where the $z_{k}$'s satisfy 
%
\[
	\mathbb{E}\left[z_{k}\mymid t_{k},\cdots,t_{1}, \bm{V}_{t_{k}},\cdots,\bm{V}_{t_{1}}\right]=0.
\]
%

We intend to invoke the Freedman inequality 
to control $X_K$ for any $K$ obeying $K\leq T$.
Similar to the synchronous counterpart, we can see that
%
\begin{align*}
	B & \coloneqq \max_{1<k\leq K}\|z_{k}\|_{\infty} \leq \eta_k^K \big( \big\|\bm{P}_{t_k+1} \|_1 + \|\bm{P}\|_1 \big) \big\|\bm{V}_{t_k}\|_{\infty} \leq \frac{2\eta_k^K}{1-\gamma} \leq \frac{2\eta}{1-\gamma} , \\
W & \coloneqq\sum_{k=K_{\beta}}^{K}\mathsf{Var}\big(z_{k}\mymid t_{k},\cdots,t_{1},\bm{V}_{t_{k}},\cdots,\bm{V}_{t_{1}}\big)=\gamma^{2}\sum_{k=K_{\beta}}^{K}\big(\eta_{k}^{(K)}\big)^{2}\mathsf{Var}\big((\bm{P}_{t_{k}+1}-\bm{P})\bm{V}_{t_{k}}\mid\bm{V}_{t_{k}}\big)\\
 & \leq\sum_{k=K_{\beta}}^{K}\big(\eta_{k}^{(K)}\big)^{2}\mathsf{Var}_{\bm{P}(s,a)}\big(\bm{V}_{t_{k}}\big)\leq\Big(\max_{K_{\beta}\leq k\leq K}\eta_{k}^{(K)}\Big)\bigg(\sum_{k=K_{\beta}}^{K}\eta_{k}^{(K)}\bigg)\mathsf{Var}_{\bm{P}(s,a)}\big(\bm{V}_{t_{k}}\big)\\
 & \leq\eta\max_{K_{\beta}\leq k\leq K}\mathsf{Var}_{\bm{P}(s,a)}\big(\bm{V}_{t_{k}}\big)\leq\eta\max_{t_{K_{\beta}}\leq i\leq t_{K}}\mathsf{Var}_{\bm{P}(s,a)}\big(\bm{V}_{i}\big) ,
\end{align*}
%
where we have made use of \eqref{eq:sum-eta-i-t-infinite}. In addition, we make note of a trivial upper bound on $W$ as follows
%
	\begin{align*}
\sigma^{2} & :=\frac{\eta}{(1-\gamma)^2} \geq \eta\max_{t_{K_{\beta}}\leq i\leq t_{K}}\mathsf{Var}_{\bm{P}(s,a)}\big(\bm{V}_{i}\big) 
		\geq W_{t}.
\end{align*}
%
With the preceding bounds in place, applying the Freedman inequality in Theorem \ref{thm:Freedman} 
and taking $L=\log_2 \frac{1}{1-\gamma} $ imply that 
%
\begin{align*}
|X_{K}| & \le\sqrt{8\max\bigg\{ W,\frac{\sigma^{2}}{2^{L}}\bigg\}\log\frac{4|\mathcal{S}||\mathcal{A}|T^{2}\log_{2}\frac{1}{1-\gamma}}{\delta}}+\frac{8\eta}{3(1-\gamma)}\log\frac{4|\mathcal{S}||\mathcal{A}|T^{2}\log_{2}\frac{1}{1-\gamma}}{\delta}\\
 & \le\sqrt{8\eta\max\Big\{\max_{t_{K_{\beta}}\leq i\leq t_{K}}\mathsf{Var}_{\bm{P}(s,a)}(\bm{V}_{i}),1\Big\}\log\frac{4|\mathcal{S}||\mathcal{A}|T^{2}\log_{2}\frac{1}{1-\gamma}}{\delta}}+\frac{8\eta}{3(1-\gamma)}\log\frac{4|\mathcal{S}||\mathcal{A}|T^{2}\log_{2}\frac{1}{1-\gamma}}{\delta}\\
 & \le\sqrt{\frac{16\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)T\mumin}\Big(\max_{t_{K_{\beta}}\leq i\leq t_{K}}\mathsf{Var}_{\bm{P}(s,a)}(\bm{V}_{i})+1\Big)}+\frac{6\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{2}T\mumin}
\end{align*}
%
with probability at least $1 - \frac{\delta}{|\mathcal{S}||\mathcal{A}|T^2}$, provided that $\eta \le \frac{\log^{3}T}{(1-\gamma)T\mumin}$.
We can thus conclude the proof by taking the union bound over all $(s, a)\in \cS \times \cA$ and all $1 \le K_{\beta} \le K \le T$. 
%
\end{proof}


SyncQlearning.tex

\documentclass[10pt,english]{article}
\usepackage[T1]{fontenc}
\usepackage[latin9]{inputenc}
\usepackage{geometry}
\geometry{verbose,tmargin=1in,bmargin=1in,lmargin=1in,rmargin=1in}
\usepackage[active]{srcltx}
\usepackage{babel}
\usepackage{verbatim}
\usepackage{mathtools}
\usepackage{bm}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage[unicode=true,
 bookmarks=false,
 breaklinks=false,pdfborder={0 0 1},colorlinks=false]
 {hyperref}
\hypersetup{
 colorlinks,linkcolor=blue,anchorcolor=blue,citecolor=blue}

\makeatletter

\usepackage{microtype,comment}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{booktabs} 
\usepackage{hyperref}
\usepackage{verbatim}
\usepackage{mathtools}
\usepackage{bm}
\usepackage{amsmath}
\usepackage{amssymb} 
\usepackage{amsthm}
\usepackage{babel}
\usepackage{cite}
\usepackage{amsthm}
\usepackage{dsfont}
\usepackage{array}
\usepackage{mathrsfs}
\usepackage{color}

\allowdisplaybreaks

\usepackage{natbib}
\usepackage{xfrac}
\usepackage{enumitem}
% \setlist[itemize]{leftmargin=1.5em}
% \setlist[enumerate]{leftmargin=1.5em}

\usepackage{babel}
\usepackage{algorithm}% http://ctan.org/pkg/algorithms
\usepackage{algorithmic}% http://ctan.org/pkg/algorithmicx
% \usepackage{algpseudocode}
\usepackage{arydshln}

% \usepackage{showlabels}
% \usepackage[inline]{showlabels}



\theoremstyle{plain} 
\newtheorem{lemma}{\textbf{Lemma}} 
\newtheorem{proposition}{\textbf{Proposition}}
\newtheorem{theorem}{\textbf{Theorem}}\setcounter{theorem}{0}
\newtheorem{corollary}{\textbf{Corollary}} 
\newtheorem{assumption}{\textbf{Assumption}}
\newtheorem{example}{\textbf{Example}} 
\newtheorem{definition}{\textbf{Definition}}
\newtheorem{fact}{\textbf{Fact}}
\newtheorem{property}{Property}
\theoremstyle{definition}
\theoremstyle{remark}
\newtheorem{condition}{Condition}
\newtheorem{claim}{Claim}
\newtheorem{conjecture}{Conjecture}
\newtheorem{remark}{\textbf{Remark}}


\newcommand{\bPi}{{\bf \Pi}}
\newcommand{\Pq}{{\bm P}^\pi}
\newcommand{\e}{{\bm e}}
\newcommand{\real}{\mathbb{R}}
\newcommand{\Var}{\mathsf{Var}}
\newcommand{\Ind}{\bm I}

\newcommand{\pib}{\pi_{\mathsf{b}}}
\newcommand{\tmix}{t_{\mathsf{mix}}}

\newcommand{\cS}{\mathcal{S}}
\newcommand{\cA}{\mathcal{A}}

\newcommand{\mymid}{\,|\,}

\newcommand{\Qhat}{\widehat{\bm Q}}
\newcommand{\Qstar}{ {\bm Q}^\star}
\newcommand{\Qhatstar}{\widehat{\bm Q}^\star}
\newcommand{\Qperstar}{Q_{\bm{r}_{\mathrm{p}}}^{\star}}
\newcommand{\Qperhatstar}{\widehat{Q}_{\bm{r}_{\mathrm{p}}}^{\star}}


\newcommand{\Vstar}{{\bm V}^\star}
\newcommand{\Vhat}{\widehat{\bm V}}
\newcommand{\Vhatstar}{\widehat{\bm V}^\star}
\newcommand{\Vhatpi}{\widehat{\bm V}^\pi}

\newcommand{\Phat}{\widehat{\bm{P}}}
\newcommand{\abPbm}{\bm{P}^{\mathsf{abs}}}
\newcommand{\bP}{\bm{P}}
\newcommand{\abPbmhat}{\widehat{\bm{P}}^{\mathsf{abs}}}

\newcommand{\Pv}{{\bm P}_{\pi}}
\newcommand{\Pvstar}{{\bm P}_{\pi^{\star}}}
\newcommand{\Phatv}{{\widehat{\bm P}}_{\pi}}
\newcommand{\abP}{P^{(s,a,u)}}
\newcommand{\ceta}{c_\eta}

\newcommand{\pistar}{{\pi}^\star}
\newcommand{\pihat}{\widehat{\pi}}
\newcommand{\pihatstar}{\widehat{\pi}^\star}

\newcommand{\rewabs}{r^{\mathsf{abs}}}
\newcommand{\rewp}{\bm{r}_{\mathrm{p}}}
\newcommand{\br}{\bm r}

\newcommand{\mumin}{\mu_{\mathsf{min}}}
\newcommand{\tcover}{t_{\mathsf{frame}}}

\newcommand{\MDP}{\mathcal{M}}

\newcommand{\defn}{\coloneqq}


\usepackage{color}
\definecolor{yxc}{RGB}{255,0,0}
\definecolor{yjc}{RGB}{0,155,155}
\definecolor{ytw}{RGB}{255,69,0}
\definecolor{gen}{RGB}{0,0,200}
\definecolor{cxc}{RGB}{0,140,0}

\newcommand{\yjc}[1]{\textcolor{yjc}{[YJC: #1]}}
\newcommand{\yxc}[1]{\textcolor{yxc}{[YXC: #1]}}
\newcommand{\yutingcomment}[1]{\textcolor{ytw}{[YTW: #1]}}
\newcommand{\gen}[1]{\textcolor{gen}{[Gen: #1]}}
\newcommand{\cxc}[1]{\textcolor{cxc}{[CXC: #1]}}

 
\begin{document}

\title{Is Q-Learning Minimax Optimal? \\ A Tight Sample Complexity Analysis} 

\author{Gen Li\thanks{Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USA.} \\
%; email: \texttt{g-li16@mails.tsinghua.edu.cn}. 
Princeton ECE  \\
	\and
	Changxiao Cai\thanks{Department of Biostatistics, University of Pennsylvania, Philadelphia, PA 19104, USA.} \\
%; email: \texttt{ccai@princeton.edu} 	
 UPenn Biostatistics \\
 	\and
	Yuxin Chen\footnotemark[1] \\
	%\thanks{Department of Electrical Engineering, Princeton University, Princeton, NJ 08544, USA.} 
%; email: \texttt{yuxin.chen@princeton.edu} 	
 Princeton ECE \\
 	\and
	Yuantao Gu\thanks{Department of Electronic Engineering, Tsinghua University, Beijing 100084, China} \\
	Tsinghua EE  \\ 
	%\thanks{Department of Electronic Engineering, Tsinghua University, Beijing 100086, China.} 
%	; email: \texttt{gyt@tsinghua.edu.cn}
  	\and 
	Yuting Wei\thanks{Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA 19104, USA.}\\
	UPenn Statistics \\
%; email: \texttt{ytwei@cmu.edu}  
	\and
	Yuejie Chi\thanks{Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA.}\\
	CMU ECE \\
%; email: \texttt{yuejiechi@cmu.edu}
	}
	
\date{February 2021; ~~ Revised: November 2021}	

\maketitle

\begin{abstract}
Q-learning, which seeks to learn the optimal Q-function of a Markov decision process (MDP) in a model-free fashion, lies at the heart of reinforcement learning. When it comes to the synchronous setting (such that independent samples for all state-action pairs are drawn from a generative model in each iteration), substantial progress has been made towards understanding the sample efficiency of Q-learning. Consider a $\gamma$-discounted infinite-horizon MDP with state space $\cS$ and action space $\cA$: to yield an entrywise $\varepsilon$-approximation of the optimal Q-function,  state-of-the-art theory for Q-learning requires a sample size exceeding
	the order of $\frac{|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^5\varepsilon^{2}}$, which fails to match existing minimax lower bounds. This gives rise to natural questions: what is the sharp sample complexity of Q-learning? Is Q-learning provably sub-optimal? This paper addresses these questions for the synchronous setting: (1) when $|\cA|=1$ (so that Q-learning reduces to TD learning), we prove that the sample complexity of TD learning is minimax optimal and scales as $\frac{|\mathcal{S}|}{(1-\gamma)^3\varepsilon^2}$ (up to log factor); (2) when $|\cA|\geq 2$, we settle the sample complexity of Q-learning to be on the order of $\frac{|\mathcal{S}||\mathcal{A}|}{(1-\gamma)^4\varepsilon^2}$ (up to log factor). Our theory unveils the strict sub-optimality of Q-learning when $|\cA|\geq 2$, and rigorizes the negative impact of over-estimation in Q-learning. Finally, we extend our analysis to accommodate asynchronous Q-learning (i.e., the case with Markovian samples), sharpening the horizon dependency of its sample complexity to be $\frac{1}{(1-\gamma)^4}$. 
\end{abstract}


	
	
%The key ingredients of our analysis lie in the establishment of novel error decompositions and recursions in establishing the upper bound, as well as construction of a hard instance that highlight the over-estimation issue of Q-learning, which might shed light on how to analyze finite-sample performance of other Q-learning variants. 


%, and (2) developing a matching lower bound to confirm the sharpness of our result. In addition, we demonstrate the near minimax-optimality of temporal difference learning (TD-learning) for policy evaluation in a Markov reward process, which corresponds to the special case of Q-learning when the action space is a singleton

%--- achievable by model-based approaches --- 


\medskip

\noindent\textbf{Keywords: }Q-learning, temporal difference learning, effective horizon, sample
complexity, minimax optimality, lower bound, over-estimation 

\setcounter{tocdepth}{2}
\tableofcontents{}


\input{introduction.tex}

\input{backgrounds.tex}

\input{results.tex}

\input{analysis.tex}

\input{AsynQ.tex}

\input{conclusions.tex}


\section*{Acknowledgements}

G.~Li and Y.~Chen are supported in part by the AFOSR YIP award FA9550-19-1-0030,
the ONR grant N00014-19-1-2120, the ARO YIP award W911NF-20-1-0097,  the NSF grants CCF-2106739, CCF-1907661, DMS-2014279, IIS-1900140 and IIS-2100158. 
Y.~Wei is supported in part by the NSF grants CCF-2007911 and DMS-2015447. 
Y.~Gu is supported in part by the grant NSFC-61971266. 
Y.~Chi is supported in part by the grants ONR N00014-18-1-2142 and N00014-19-1-2404, 
the NSF grants CCF-1806154 and CCF-2007911. 
The authors are grateful to Laixi Shi for helpful discussions about the lower bound, and thank Shaocong Ma for pointing out some errors in an early version of this work.
Part of this work was done while G.~Li, Y.~Chen and Y.~Wei were visiting the Simons Institute for the Theory of Computing.

\appendix

\input{preliminary-Freedman.tex}

\input{infinite-synchronous-Q.tex}

\input{TD-learning.tex}

\input{lowerbound.tex}



\input{AsynQ_analysis.tex}
  



\bibliographystyle{apalike}
\bibliography{bibfileRL}




\end{document}


TD-learning.tex
\section{Analysis for TD learning (Theorem~\ref{thm:policy-evaluation})}
\label{sec:TD-learning-analysis}


As it turns out, if $|\cA|=1$ (which reduces to the case of TD learning), we can further modify the previous analysis in Section~\ref{sec:Analysis:-infinite-horizon-MDPs} to yield an improved $\frac{1}{(1-\gamma)^3}$ scaling. This forms the main content of this section, which leads to the proof of Theorem~\ref{thm:policy-evaluation} for TD learning. Akin to the Q-learning case, we proceed to establish a more general version of Theorem~\ref{thm:policy-evaluation} that covers the full $\varepsilon$-range. This is formally stated below, which subsumes Theorem~\ref{thm:policy-evaluation} as a special case. 
%
\begin{theorem}
	\label{thm:policy-evaluation-general}
	Consider any $\gamma\in(0,1)$ and any $\varepsilon \in \big( 0, \frac{1}{1-\gamma} \big]$. Theorem~\ref{thm:policy-evaluation} continues to hold if 
	%
	\begin{equation}
		T\ge \frac{c_{3}\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{3}\min\{\varepsilon,\varepsilon^{2}\}}
		\label{eq:thm:sample-size-TD}
	\end{equation}
	%
	for some sufficiently large universal constant $c_3>0$. 
\end{theorem}
%
 

% In this section, we establish our theoretical guarantees for TD learning as stated in Theorem~\ref{thm:policy-evaluation}. 


\subsection{Preliminary facts}
\label{sec:infinite_prelim_TD}

Before embarking on the analysis, we begin by presenting several useful preliminary facts. 
The first one is a direct
consequence of the claimed iteration complexity (\ref{eq:thm:sample-size-TD})
when $\varepsilon\leq\frac{1}{1-\gamma}$: 
%
\begin{equation}
	T\geq \frac{c_{3}\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{3}\min\{\varepsilon,\varepsilon^{2}\}}
	\geq\frac{c_{3}\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{2}},
	\label{eq:T-bound-TD}
\end{equation}
%
a simple fact that will be used multiple times. 
In addition, the update rule \eqref{eqn:td-learning} of TD learning can be expressed using vector/matrix notation as follows
%
\begin{align} 
\label{eq:iteration-rule-td}
	\bm V_t = (1-\eta_t)\bm V_{t-1} + \eta_t(\bm r + \gamma \bm P_t\bm V_{t-1}) 
	\qquad \text{for all } t\geq 1,
\end{align}
%
where the matrix $\bm{P}_t\in \{0,1\}^{|\cS|\times |\cS|}$ obeys
%
\[
	\bm{P}_t(s,s') \coloneqq \begin{cases} 1, \quad &\text{if }s'=s_t(s) \\ 0, & \text{else} \end{cases}
\]
%
for any $s,s'\in \cS$. 
In the sequel, we collect a few other facts concerning the range of $\bm{V}_{t}$ and learning rates.



\paragraph{Range of $\bm{V}_{t}$.} 


We claim that: when the initialization $\bm{V}_{0}$ obeys $\bm{0} \leq \bm{V}_{0}\leq \frac{1}{1-\gamma} \bm{1}$, 
the TD learning iterates obey
%
\begin{align} \label{eq:V-i-LB-UB}
	\bm{0}\le\bm{V}_{t}\le\frac{1}{1-\gamma}\bm{1}  \qquad \text{and} 
	\qquad \|\bm{V}_{t}-\bm{V}^{\star}\|_{\infty}\le\frac{1}{1-\gamma}
	\qquad \text{for all }t\geq 0, 
\end{align}
%
provided that $0\leq \eta_t \leq 1$ for all $t\geq 0$. The proof follows immediately by repeating the proof of Lemma~\ref{lemma:non-negativity-Qt-Vt} (see Section~\ref{sec:infinite_prelim}) with $|\cA|=1$, and is hence omitted for brevity. 

%
\begin{comment}
\begin{proof}
We prove this by induction. First,
the base case with $t=0$ satisfies (\ref{eq:V-i-LB-UB}) given our assumptions. 
%
Next, suppose that (\ref{eq:V-i-LB-UB}) is valid for the $(t-1)$-th
iteration, 
%
\begin{equation}
	\bm{0}\le\bm{V}_{t-1}\le\frac{1}{1-\gamma}\bm{1}. 
	\label{eq:induction-hypothesis-TD-V-1}
\end{equation}
%
Given the basic facts
that $\bm{0}\leq\bm{r}\leq\bm{1}$, $\bm{P}_{t}\geq\bm{0}$ and $\|\bm{P}_{t}\|_{1}=1$,
we can invoke the update rule (\ref{eq:iteration-rule})
and the induction hypothesis \eqref{eq:induction-hypothesis-TD-V-1} to demonstrate that
%
\[
\bm{V}_{t}=(1-\eta_{t})\bm{V}_{t-1}+\eta_{t}(\bm{r}+\gamma\bm{P}_{t}\bm{V}_{t-1})\geq\bm{0} 
\]
%
and
%
\begin{align*}
\bm{V}_{t} & =(1-\eta_{t})\bm{V}_{t-1}+\eta_{t}(\bm{r}+\gamma\bm{P}_{t}\bm{V}_{t-1})\\
 & \leq(1-\eta_{t})\,\|\bm{V}_{t-1}\|_{\infty}\bm{1}+\eta_{t}\big(\|\bm{r}\|_{\infty}+\gamma\|\bm{P}_{t}\|_{1}\|\bm{V}_{t-1}\|_{\infty}\big)\bm{1}\\
 & \leq(1-\eta_{t})\frac{1}{1-\gamma}\bm{1}+\eta_{t}\Big(1+\frac{\gamma}{1-\gamma}\Big)\bm{1}=\frac{1}{1-\gamma}\bm{1}. 
\end{align*}
%
This establishes (\ref{eq:V-i-LB-UB}) for the $t$-th iteration,
thus concluding the proof via an induction argument.
%
\end{proof}
\end{comment}

%Together with the fact $\bm{0}\le\bm{V}^{\star}\le\frac{1}{1-\gamma}\bm{1}$,
%we have
%%
%\begin{equation}
%\|\bm{V}_{t}-\bm{V}^{\star}\|_{\infty}\le\frac{1}{1-\gamma}\qquad\text{for all }t\geq0, \label{eq:Delta-t-range}
%\end{equation}
%%
%which also confirms that $0\leq \varepsilon\leq\frac{1}{1-\gamma}$ is the full $\varepsilon$-range we need to consider. 
%




\paragraph{Learning rates.} 


We shall also collect several useful results concerning the learning rates $\{\eta_{t}\}$.
Let us abuse the notation by defining the following crucial quantities:
%
\begin{equation}
\eta_{k}^{(t)}\coloneqq\begin{cases}
\prod_{i=1}^{t}\big(1-\eta_{i}(1-\gamma) \big), & \text{if }k=0,\\
\eta_{k}\prod_{i=k+1}^{t} \big( 1-\eta_{i}(1-\gamma) \big), & \text{if }0<k<t,\\
\eta_{t}, & \text{if }k=t. 
\end{cases}\label{def:eta-k-t}
\end{equation}
%
Note that this definition \eqref{def:eta-k-t} differs from the one \eqref{def:eta-i-t} used for Q-learning, and will only be employed in this section. 
Consider any iteration number $t$ satisfying
%
\begin{equation}
t\ge\frac{T}{c_{2}\log T}. \label{eq:t-LB-c2}
\end{equation}
%
Clearly, the learning rate $\eta_t$ under Assumption~\eqref{eq:thm:eta-TD} obeys
%
\begin{equation}
(1-\gamma)\eta_{t}\geq\frac{1-\gamma}{1+\frac{c_{1}(1-\gamma)T}{\log^{2}T}}\geq\frac{1-\gamma}{\frac{2c_{1}(1-\gamma)T}{\log^{2}T}}=\frac{\log^{2}T}{2c_{1}T}.
	\label{eq:eta-t-lower-bound-123}
\end{equation}
%
In what follows, we intend to bound $\eta_{k}^{(t)}$ for two cases separately.
%
\begin{itemize}
%
\item For any $i$ obeying $0\leq i\le t/2$, it is easily seen from \eqref{eq:eta-t-lower-bound-123} that
%
\begin{subequations}\label{eq:eta-i-t-UB12-TD}
%
\begin{align}
\eta_{i}^{(t)} & \le\big(1-\eta_{t/2}(1-\gamma)\big)^{t/2}\le\Big(1-\frac{\log^{2}T}{2c_{1}T}\Big)^{t/2}\leq\Big(1-\frac{\log^{2}T}{2c_{1}T}\Big)^{\frac{T}{2c_{2}\log T}} \notag\\
 & =\left\{ \Big(1-\frac{\log^{2}T}{2c_{1}T}\Big)^{\frac{2c_{1}T}{\log^{2}T}}\right\} ^{\frac{\log T}{4c_{1}c_{2}}}\leq\frac{1}{T^{2}},
	\label{eq:eta-i-t-UB1-TD}
\end{align}
%
where the last inequality holds as long as $c_{1}c_{2}\le 1/8$ and \eqref{eq:T-bound-TD} holds. 


\item When it comes to the case with $i> t/2$, we can develop the following upper bound
%
\begin{equation}
\eta_{i}^{(t)}\le\eta_{i}\leq\frac{1}{c_{2}(1-\gamma)i/\log^{2}T}<\frac{2\log^{3}T}{(1-\gamma)T},\label{eq:eta-i-t-UB2-TD}
\end{equation}
%
which relies on Assumption~\eqref{eq:thm:eta-TD}. 
\end{subequations}
%
\end{itemize} 
%
In addition, 
%we note that
%%
%\[
%	\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)=\prod_{i=k+1}^{t}\big((1-\eta_{i})\bm{I}+\eta_{i}\gamma\bm{P}\big)
%\]
%%
%is a non-negative matrix as long as $0\leq \eta_i\leq 1$. 
given that $\bm{P}^k\bm{1}=\bm{1}$ for any integer $k>0$, it can be easily verified that
%
\[
	\prod_{i = k+1}^t \big(\bm{I}-\eta_i(\bm{I}-\gamma\bm P)\big) \bm{1} = \prod_{i = k+1}^t \big(1-\eta_i(1-\gamma)\big) \bm{1} ,
\]
%
and as a result,
%
\begin{align}
	\Bigg\|\prod_{i = k+1}^t \big(\bm{I}-\eta_i(\bm{I}-\gamma\bm P)\big)\Bigg\|_1 =  \prod_{i = k+1}^t \big(1-\eta_i(1-\gamma)\big).
	\label{eq:eta-prod-complex-formula}
\end{align}
%



\subsection{Proof of Theorem~\ref{thm:policy-evaluation-general}} 


\paragraph{Step 1: decomposing the error $\bm V_t - \bm V^{\star}$. }

Taking $\bm \Delta_t \defn \bm V_t - \bm V^{\star}$, via the basic relation \eqref{eq:iteration-infinite}, the TD learning update rule can be written as 
% (\ref{eq:iteration-rule-td}) as follows
%
\begin{align}
\bm \Delta_t 
% &= (1-\eta_t)\bm \Delta_{t-1} + \eta_t(\bm r + \gamma \bm P_t\bm V_{t-1} - \bm V^{\star}) \nonumber\\
	% &= (1-\eta_t)\bm \Delta_{t-1} + \eta_t(\bm r + \gamma \bm P_t\bm V_{t-1} - \bm{r} - \gamma \bm{P} \bm{V}^{\star} ) \nonumber\\
&= (1-\eta_t)\bm \Delta_{t-1} + \eta_t\gamma \big(\bm P\bm \Delta_{t-1} + (\bm P_t - \bm P)\bm V_{t-1} \big) \notag\\
	&= \big(\bm{I}-\eta_t(\bm{I}-\gamma\bm{P})\big)\bm \Delta_{t-1} + \eta_t\gamma  (\bm P_t - \bm P)\bm V_{t-1} .
\end{align}
%
%where $\leq$ denotes entrywise comparison. 
% , and we have used the Bellman equation $\bm V^{\star} = \bm r + \gamma \bm P\bm V^{\star}$.
Invoking the above relation recursively then leads to
%
\begin{align}
\bm \Delta_t = \prod_{i = 1}^t \big(\bm{I}-\eta_i(\bm{I}-\gamma\bm P)\big)\bm \Delta_0 + \underbrace{\sum_{k = 1}^t\eta_k\prod_{i = k+1}^t \big(\bm{I}-\eta_i(\bm{I}-\gamma\bm P)\big)\gamma(\bm P_k - \bm P)\bm V_{k-1}}_{ \eqqcolon\, \bm \xi_t}.
	\label{eq:defn-xit-TD}
\end{align}
%


%Consider the case with constant learning rates $\eta_t \equiv \eta$.
%It follows that
%%
%\begin{align}
% & \sum_{k=1}^{t}\eta_{k}\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\gamma(\bm{P}_{k}-\bm{P})\bm{V}_{k-1}\notag\\
% & \qquad=\sum_{k=1}^{t}\eta\big((1-\eta)\bm{I}+\eta\gamma\bm{P}\big)^{t-k}\gamma(\bm{P}_{k}-\bm{P})\bm{V}_{k-1}\notag\\
% & \qquad=\sum_{k=1}^{t}\sum_{i=0}^{t-k}\tbinom{t-k}{i}\eta(1-\eta)^{t-k-i}\big(\eta\gamma\bm{P}\big)^{i}\gamma(\bm{P}_{k}-\bm{P})\bm{V}_{k-1}\notag\\
% & \qquad=\sum_{i=0}^{t-1}\big(\gamma\bm{P}\big)^{i}\sum_{k=1}^{t-i}\tbinom{t-k}{i}\eta^{i+1}(1-\eta)^{t-k-i}\gamma(\bm{P}_{k}-\bm{P})\bm{V}_{k-1},
%\end{align}
%%
%which leads to
%\begin{align}
%|\bm{\xi}_{t}| \lesssim \max_{1\le i \le t-1}\Big(\sum_{k = 1}^{t-i}\big[\tbinom{t-k}{i}\eta^{i+1}(1-\eta)^{t-k-i}\big]^2\Big)^{1/2}\big(\Ind - \gamma \bm P\big)^{-1}\sqrt{\mathsf{Var}_{\bm{P}}\big(\bm{V}^{\star}\big)}
%\end{align}
%In addition, for $\overline{\bm V}_t = \frac{1}{t}\sum_{j = 0}^{t-1} \bm V_j$ and $\overline{\bm \Delta}_t \defn \overline{\bm V}_t - \bm V^{\star} = \frac{1}{t}\sum_{j = 0}^{t-1} \bm \Delta_j$, one has
%\begin{align}
%\overline{\bm \Delta}_t &= \frac{1}{t}\sum_{j = 0}^{t-1} \prod_{i = 1}^j \big(1-\eta_i(1-\gamma\bm P)\big)\bm \Delta_0 + \frac{1}{t}\sum_{j = 0}^{t-1} \sum_{k = 1}^j\eta_k\prod_{i = k+1}^j \big(1-\eta_i(1-\gamma\bm P)\big)\gamma(\bm P_k - \bm P)\bm V_{k-1} \\
%&= \frac{1}{t}\sum_{j = 0}^{t-1} \prod_{i = 1}^j \big(1-\eta_i(1-\gamma\bm P)\big)\bm \Delta_0 + \underbrace{\frac{1}{t}\sum_{k = 1}^{t-1}\Big[\sum_{j = k}^{t-1} \eta_k\prod_{i = k+1}^j \big(1-\eta_i(1-\gamma\bm P)\big)\Big]\gamma(\bm P_k - \bm P)\bm V_{k-1}}_{\bm \zeta_t}.
%\end{align}


\paragraph{Step 2: controlling the first term of \eqref{eq:defn-xit-TD}.} 
With regards to the first term  of \eqref{eq:defn-xit-TD}, we make the observation that
%
\begin{align}
\Big\Vert \prod_{i=1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\bm{\Delta}_{0}\Big\Vert _{\infty} 
& \le \Big\Vert \prod_{i=1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\Big\Vert _{1}\left\Vert \bm{\Delta}_{0}\right\Vert _{\infty} \notag\\
 & =\left\{ \prod_{i=1}^{t}\big(1-\eta_{i}(1-\gamma)\big)\right\} \left\Vert \bm{\Delta}_{0}\right\Vert _{\infty} \notag\\
 & \le\eta_{0}^{(t)}\cdot\frac{1}{1-\gamma}\le\frac{1}{(1-\gamma)T^{2}}, 
\end{align}
%
where the second line arises from \eqref{eq:eta-prod-complex-formula}, 
and the last inequality holds true due to \eqref{eq:eta-i-t-UB1-TD} as long as $t\ge\frac{T}{c_{2}\log T}.$


\paragraph{Step 3: controlling the second term of \eqref{eq:defn-xit-TD}.} 

We then move on to the second term $\bm \xi_t$ in \eqref{eq:defn-xit-TD}, which admits the following expression
%
\begin{align}
	\bm \xi_t = \sum_{k = 1}^t\bm{z}_{k}\qquad\text{with }\bm{z}_{k}\coloneqq\eta_k\prod_{i = k+1}^t \big(\bm{I}-\eta_i(\bm{I}-\gamma\bm P)\big)\gamma(\bm P_k - \bm P)\bm V_{k-1}. 
	\label{eq:defn-xit-zk-TD}
\end{align}
%
Here, the summands $\{\bm{z}_{k}\}$ clearly satisfy
%
\[
\mathbb{E}\big[\bm{z}_{k}\mymid\bm{V}_{k-1},\cdots,\bm{V}_{0}\big]=\bm{0}.
\]
%
We then attempt to invoke the Freedman inequality (see Theorem~\ref{thm:Freedman})
to control this term.
Towards this end, there are several quantities that need to be calculated.  
%
\begin{itemize}
\item First of all, we observe that
\begin{align}
B & \coloneqq \max_{1\le k\leq t} \|\bm{z}_{k}\|_{\infty}\leq\max_{1\le k \leq t} \Big\|\eta_k\prod_{i = k+1}^t \big(\bm{I}-\eta_i(\bm{I}-\gamma\bm P)\big)\gamma(\bm P_k - \bm P)\bm V_{k-1} \Big\|_{\infty} \nonumber\\
 & \leq\max_{1\le k \leq t} \Big\|\eta_k\prod_{i = k+1}^t \big(\bm{I}-\eta_i(\bm{I}-\gamma\bm P)\big) \Big\|_{1}  \big\| (\bm P_k - \bm P)\bm V_{k-1} \big\|_{\infty} \nonumber\\
	& =\max_{1\le k \leq t} \left\{ \eta_k\prod_{i = k+1}^t \big(1-\eta_i(1-\gamma)\big) \right\} 
	\big\| (\bm P_k - \bm P)\bm V_{k-1} \big\|_{\infty} \nonumber\\	
 & \leq\max_{1\le k \leq t}\eta_{k}^{(t)}\big(\|\bm{P}_{k}\|_{1}+\|\bm{P}\|_{1}\big)\|\bm{V}_{k-1}\|_{\infty}\leq\frac{4\log^{3}T}{(1-\gamma)^2T}, \label{eq:TD_B}
\end{align}
%
where the third line again makes use of the relation \eqref{eq:eta-prod-complex-formula}
and the last line follows the facts $\|\bm{P}_{k}\|_{1}=\|\bm{P}\|_{1}=1$, $\|\bm{V}_{k-1}\|_{\infty} \le 1/(1-\gamma)$,
as well as the properties \eqref{eq:eta-i-t-UB12-TD}. 


\item The next step is to control certain variance terms.  
Towards this, we first make note of a userful fact.  
For any given non-negative vector $\bm{u}=[u_{i}]_{1\leq i\leq|\cS|}\geq\bm{0}$
and any vector $\bm{v}$, it is easily seen that
%
\begin{align}
\mathsf{Var}\Big(\bm{u}^{\top}(\bm{P}_{k}-\bm{P})\bm{v}\Big) & =\sum_{i=1}^{|\cS|}u_{i}^{2}\mathsf{Var}\Big((\bm{P}_{k}-\bm{P})_{i,\cdot}\bm{v}\Big)\leq\Big\{\max_{i}|u_{i}|\Big\}\big[u_{1},\cdots,u_{|\cS|}\big]\mathsf{Var}_{\bm{P}}\left(\bm{v}\right)\nonumber \\
	& \leq \|\bm{u}\|_1 \bm{u}^{\top}\mathsf{Var}_{\bm{P}}\left(\bm{v}\right),\label{eq:Var-u-Pk-v-bound}
\end{align}
%
where we remind the reader of the notation $\mathsf{Var}_{\bm{P}}(\bm{v})$ in \eqref{eq:defn-Var-P-V}. 
Additionally, 
for any vector $\bm{a}=[a_j]$, let us employ the notation $\mathsf{Var}\big(\bm{a}\mymid\bm{V}_{k-1},\cdots,\bm{V}_{0}\big)$
to represent a vector whose $j$-th entry is given by $\mathsf{Var}\big(a_{j}\mymid\bm{V}_{k-1},\cdots,\bm{V}_{0}\big)$.
Armed with this notation, we obtain
%
\begin{align}
\mathsf{Var}\big(\bm{z}_{k}\mymid\bm{V}_{k-1},\cdots,\bm{V}_{0}\big) 
& \leq\gamma^{2}\Big\Vert \eta_{k}\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\Big\Vert _{1}\left\{ \eta_{k}\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\right\} \mathsf{Var}_{\bm{P}}\left(\bm{V}_{k-1}\right) \notag\\
 & =\gamma^{2}\left\{ \eta_{k}\prod_{i=k+1}^{t}\big(1-\eta_{i}(1-\gamma)\big)\right\} \left\{ \eta_{k}\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\right\} \mathsf{Var}_{\bm{P}}\left(\bm{V}_{k-1}\right) \notag\\
 & \leq \eta_{k}\eta_{k}^{(t)}\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\mathsf{Var}_{\bm{P}}\left(\bm{V}_{k-1}\right),
\end{align}
%
where the first inequality is a consequence of  \eqref{eq:Var-u-Pk-v-bound} and the definition of $\bm{z}_k$ (cf.~\eqref{eq:defn-xit-zk-TD}), the second line arises from \eqref{eq:eta-prod-complex-formula}, and the last relation results from the definition of $\eta_k^{(t)}$. 
This in turn allows us to compute 
%
\begin{align}
\bm{W}_{t} & \coloneqq\sum_{k=1}^{t}\mathsf{Var}\big(\bm{z}_{k}\mymid\bm{V}_{k-1},\cdots,\bm{V}_{0}\big)\le\sum_{k=1}^{t}\eta_{k}\eta_{k}^{(t)}\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\mathsf{Var}_{\bm{P}}\left(\bm{V}_{k-1}\right)\nonumber\\
%
& \le \sum_{k=1}^{t/2}\eta_{k}^{(t)}\Big\Vert \eta_{k}\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\Big\Vert _{1}\|\bm{V}_{k-1}\|_{\infty}^{2}  \bm{1}
	+\sum_{k=t/2+1}^{t}\eta_{k}\eta_{k}^{(t)}\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\mathsf{Var}_{\bm{P}}\left(\bm{V}_{k-1}\right)\nonumber\\
 & \leq\sum_{k=1}^{t/2}\big(\eta_{k}^{(t)}\big)^{2}\frac{1}{(1-\gamma)^{2}} \bm{1} 
	+ \Big\{ \max_{k:\,t/2<k\leq t} \eta_{k}^{(t)} \Big\}  
	\sum_{k=t/2+1}^{t}\eta_{k}\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\mathsf{Var}_{\bm{P}}\left(\bm{V}_{k-1}\right)\nonumber\\
 & \leq\frac{1}{2(1-\gamma)^{2}T^{3}} \bm{1} +\frac{2\log^{3}T}{(1-\gamma)T} \Bigg(\sum_{k=t/2+1}^{t}\eta_{k}\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\Bigg)\max_{k:\,t/2\leq k<t}\mathsf{Var}_{\bm{P}}\big(\bm{V}_{k}\big)\nonumber\\
 & \leq\frac{1}{2(1-\gamma)^{2}T^{3}} \bm{1}
	+\frac{2\log^{3}T}{(1-\gamma)T}(\bm{I}-\gamma\bm{P})^{-1}\max_{k:\,t/2\leq k<t}\mathsf{Var}_{\bm{P}}\big(\bm{V}_{k}\big),
	\label{eq:Wt-bound-Var}
\end{align}
%
where the penultimate inequality results from \eqref{eq:eta-i-t-UB12-TD}; 
to see why the last inequality holds, observe that
%
\begin{align*}
&\sum_{k=t/2+1}^{t}\eta_{k}\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big) \\
&\qquad\qquad= (\bm{I}-\gamma\bm{P})^{-1}\sum_{k=t/2+1}^{t}\eta_{k}(\bm{I}-\gamma\bm{P})\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big) \\
&\qquad\qquad= (\bm{I}-\gamma\bm{P})^{-1}\sum_{k=t/2+1}^{t}\bigg[\prod_{i=k+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big) - \prod_{i=k}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big)\bigg] \\
&\qquad\qquad= (\bm{I}-\gamma\bm{P})^{-1} - (\bm{I}-\gamma\bm{P})^{-1}\prod_{i=t/2+1}^{t}\big(\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big) \le (\bm{I}-\gamma\bm{P})^{-1},
\end{align*}
%
where we have used the fact that all entries of $(\bm{I}-\gamma\bm{P})^{-1}$ and $\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})$ are non-negative.
 
 
\item In addition, we also derive the following
trivial upper bound based on \eqref{eq:Wt-bound-Var}:
%
\begin{align}
\big|\bm{W}_{t}\big| & \leq\frac{1}{2(1-\gamma)^{2}T^{3}}\bm{1}+\frac{2\log^{3}T}{(1-\gamma)T}\big\|(\bm{I}-\gamma\bm{P})^{-1}\big\|_{1}\max_{k:\,t/2\leq k<t}\big\|\mathsf{Var}_{\bm{P}}\big(\bm{V}_{k}\big)\big\|_{\infty}\bm{1} \notag\\
 & \leq\frac{1}{2(1-\gamma)^{2}T^{3}}\bm{1}+\frac{2\log^{3}T}{(1-\gamma)^{4}T}\bm{1}\leq\frac{3\log^{3}T}{(1-\gamma)^{4}T}\bm{1}\eqqcolon\sigma^{2}\bm{1},
	\label{eq:Wt-trivial-UB-TD}
\end{align}
%
where we have invoked the fact that $\|(\bm{I}-\gamma \bm{P})^{-1} \|_1 = 1/(1-\gamma)$. 
Therefore, by setting $K=\big\lceil 2\log_2\frac{1}{1-\gamma}\big\rceil $, one arrives at
%
\begin{equation}
\frac{\sigma^{2}}{2^{K}}\leq\frac{3\log^{3}T}{(1-\gamma)^2T}.\label{eq:sigma-2K-UB}
\end{equation}
%
\end{itemize}
Equipped with the preceding bounds, let us apply the Freedman inequality in
Theorem~\ref{thm:Freedman} and invoke the union bound over all entries
of $\bm{\xi}_{t}$ to show that 
%
\begin{align*}
|\bm{\xi}_{t}| & \le
\sqrt{8\Big(\bm{W}_{t}+\frac{\sigma^{2}}{2^{K}}\bm{1}\Big)\log\frac{8|\mathcal{S}|T\log\frac{1}{1-\gamma}}{\delta}}+\Big(\frac{4}{3}B\log\frac{8|\mathcal{S}|T\log\frac{1}{1-\gamma}}{\delta}\Big) \bm{1}\\
 & \le\sqrt{16\Big(\bm{W}_{t}+\frac{3\log^{3}T}{(1-\gamma)^2T}\bm{1}\Big)\log\frac{|\mathcal{S}|T}{\delta}}+\Big(3B\log\frac{|\mathcal{S}|T}{\delta}\Big) \bm{1}\\
	& \le\sqrt{\frac{32\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}|T}{\delta}\big)}{(1-\gamma)T}\Big((\bm{I}-\gamma\bm P)^{-1}\max_{k:\,t/2\leq k<t}\mathsf{Var}_{\bm{P}}\big(\bm{V}_{k}\big)+\frac{2}{1-\gamma}\bm{1}\Big)}+\frac{12\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}|T}{\delta}\big)}{(1-\gamma)^{2}T}\bm{1}
\end{align*}
%
with probability at least $1-\delta/T.$ 
Here, the second line follows since 
$$\log\frac{8|\mathcal{S}|T\log\frac{1}{1-\gamma}}{\delta} \le 2\log\frac{|\mathcal{S}|T}{\delta}$$
as long as $\frac{|\mathcal{S}|T}{\delta} \ge 8\log\frac{1}{1-\gamma}$, 
whereas the last line holds by using \eqref{eq:TD_B}, \eqref{eq:Wt-bound-Var} and \eqref{eq:sigma-2K-UB}.
Further, we make the observation that
%
\begin{align*}
(\bm{I}-\gamma\bm{P})^{-1}\mathsf{Var}_{\bm{P}}\big(\bm{V}^{\star}\big) & =(\bm{I}-\gamma\bm{P})^{-1}\Big(\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-(\bm{P}\bm{V}^{\star})\circ(\bm{P}\bm{V}^{\star})\Big)\\
 & =(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-\frac{1}{\gamma^{2}}(\bm{V}^{\star}-\bm{r})\circ(\bm{V}^{\star}-\bm{r})\right)\\
 & \le(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-\frac{1}{\gamma^{2}}\bm{V}^{\star}\circ\bm{V}^{\star}+\frac{2}{\gamma^{2}}\bm{r}\circ\bm{V}^{\star}\right)\\
 & \le(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-\frac{1}{\gamma}\bm{V}^{\star}\circ\bm{V}^{\star}+\frac{2}{\gamma^{2}}\bm{r}\circ\bm{V}^{\star}\right)\\
 & = \frac{1}{\gamma} (\bm{I}-\gamma\bm{P})^{-1}\left(\gamma\bm{P}-\bm{I}\right)\left(\bm{V}^{\star}\circ\bm{V}^{\star}\right)+\frac{2}{\gamma^{2}}(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{r}\circ\bm{V}^{\star}\right)\\
 % & =-\frac{1}{\gamma}\left(\bm{V}^{\star}\circ\bm{V}^{\star}\right)+\frac{2}{\gamma^{2}}(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{r}\circ\bm{V}^{\star}\right)\\
 & \leq\frac{2}{\gamma^{2}}(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{r}\circ\bm{V}^{\star}\right)\leq\frac{2}{\gamma^{2}(1-\gamma)^{2}}\bm{1},
\end{align*}
%
where the second line makes use of the basic relation $\bm{V}^{\star} = \bm{r} + \gamma\bm{P}\bm{V}^{\star}$.
As a consequence, we conclude 
\begin{align}
	(\bm{I}-\gamma\bm P)^{-1}\max_{k:\,t/2\le k<t} \mathsf{Var}_{\bm{P}}\big(\bm{V}_{k}\big) &\le (\bm{I}-\gamma\bm P)^{-1}\Big(\mathsf{Var}_{\bm{P}}\big(\bm{V}^{\star}\big) + \frac{4}{1-\gamma}\max_{k:\,t/2\le k<t} \|\bm{\Delta}_{k}\|_{\infty}\bm{1}\Big) \nonumber\\
&\le \frac{2}{\gamma (1-\gamma)^2}\Big(1 + 2\max_{k:\,t/2\le k<t}\|\bm{\Delta}_{k}\|_{\infty}\Big) \bm{1}. 
\end{align}
%
Here, the first inequality arises from \eqref{eq:Var-P-Var-star-gap-inf}, while the second inequality holds due to the facts that $\big\|(\bm{I}-\gamma\bm P)^{-1}\big\|_1 = 1/(1-\gamma)$. 
% and the following crucial property
%%
%\begin{align*}
%(\bm{I}-\gamma\bm{P})^{-1}\mathsf{Var}_{\bm{P}}\big(\bm{V}^{\star}\big) & =\sum_{k=0}^{\infty}(\gamma\bm{P})^{k}\Big(\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-(\bm{P}\bm{V}^{\star})\circ(\bm{P}\bm{V}^{\star})\Big)\\
% & =\sum_{k=0}^{\infty}(\gamma\bm{P})^{k}\left(\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-\frac{1}{\gamma^{2}}(\bm{V}^{\star}-\bm{r})\circ(\bm{V}^{\star}-\bm{r})\right)\\
% & \le\sum_{k=0}^{\infty}(\gamma\bm{P})^{k}\left(\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-\frac{1}{\gamma}\bm{V}^{\star}\circ\bm{V}^{\star}+2\bm{r}\circ\bm{V}^{\star}\right)\\
% & =\frac{1}{\gamma}\sum_{k=0}^{\infty}(\gamma\bm{P})^{k+1}\left(\bm{V}^{\star}\circ\bm{V}^{\star}\right)-\frac{1}{\gamma}\sum_{k=0}^{\infty}(\gamma\bm{P})^{k}\left(\bm{V}^{\star}\circ\bm{V}^{\star}\right)+\sum_{k=0}^{\infty}(\gamma\bm{P})^{k}\left(2\bm{r}\circ\bm{V}^{\star}\right)\\
% & =-\frac{1}{\gamma}\bm{V}^{\star}\circ\bm{V}^{\star}+(\bm{I}-\gamma\bm{P})^{-1}\left(2\bm{r}\circ\bm{V}^{\star}\right)\\
% & \le(\bm{I}-\gamma\bm{P})^{-1}(2\bm{r}\circ\bm{V}^{\star})\le\frac{2}{(1-\gamma)^{2}},
%\end{align*}
%
%\begin{align*}
%(\bm{I}-\gamma\bm{P})^{-1}\mathsf{Var}_{\bm{P}}\big(\bm{V}^{\star}\big) & =(\bm{I}-\gamma\bm{P})^{-1}\Big(\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-(\bm{P}\bm{V}^{\star})\circ(\bm{P}\bm{V}^{\star})\Big)\\
% & =(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-\frac{1}{\gamma^{2}}(\bm{V}^{\star}-\bm{r})\circ(\bm{V}^{\star}-\bm{r})\right)\\
% & \leq(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-\frac{1}{\gamma}(\bm{V}^{\star}-\bm{r})\circ(\bm{V}^{\star}-\bm{r})\right)\\
% & \le(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-\frac{1}{\gamma}\bm{V}^{\star}\circ\bm{V}^{\star}+\frac{2}{\gamma}\bm{r}\circ\bm{V}^{\star}\right)\\
% & =(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{P}-\frac{1}{\gamma}\bm{I}\right)\left(\bm{V}^{\star}\circ\bm{V}^{\star}\right)+\frac{2}{\gamma}(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{r}\circ\bm{V}^{\star}\right)\\
% & =-\frac{1}{\gamma}\left(\bm{V}^{\star}\circ\bm{V}^{\star}\right)+\frac{2}{\gamma}(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{r}\circ\bm{V}^{\star}\right)\\
% & \leq\frac{2}{\gamma}(\bm{I}-\gamma\bm{P})^{-1}\left(\bm{r}\circ\bm{V}^{\star}\right)\leq\frac{2}{\gamma(1-\gamma)^{2}}\bm{1},
%\end{align*}
%



\paragraph{Step 4: putting everything together.}

Consequently, substituting the bounds in Steps 2-3 into \eqref{eq:defn-xit-TD} yields
%
\begin{align}
 \|\bm{\Delta}_{t}\|_{\infty} & \leq 30 \sqrt{\frac{\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{3}T}\bigg(1+\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\bigg)}\qquad\text{for all}\quad t\ge\frac{T}{c_{2}\log T}. \label{eq:Delta-t-inf-recursion}
 \end{align}
%
% with the proviso that $\gamma \geq 1/2$.
Repeating the same argument as in Section~\ref{sec:combine_ub_lb}, we see that 
%
\begin{align}
\|\bm{\Delta}_{T}\|_{\infty} & \leq c_{9}\bigg(\sqrt{\frac{\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{3}T}}+\frac{\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{3}T}\bigg)
	\label{eq:Delta-T-UB-TD-123}
\end{align}
%
holds with probability at least $1-\delta$, where $c_9>0$ is some universal constant.  
As a result, one has 
%
\begin{align*}
	\|\bm{\Delta}_{T}\|_{\infty} & \leq\frac{1}{2}\bigg(\sqrt{\min\{\varepsilon,\varepsilon^{2}\}}+\min\{\varepsilon,\varepsilon^{2}\}\bigg)=\frac{1}{2}\left(\varepsilon+\varepsilon^{2}\right)\mathds{1}\{\varepsilon\leq1\}+\frac{1}{2}\left(\varepsilon+\varepsilon^{2}\right)\mathds{1}\{\varepsilon>1\}\leq\varepsilon,
\end{align*}
as long as the sample size satisfies the following 
%
\[
\frac{\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}|T}{\delta}\big)}{\gamma^2 (1-\gamma)^{3}T}\leq\frac{\min\{\varepsilon,\varepsilon^{2}\}}{c_{3}},
\]
for some constant $c_{3} \geq \max\{1, 2c_9\}$. This requirement is equivalent to condition \eqref{eq:thm:sample-size-TD} as claimed. 




analysis.tex
\section{Key analysis ideas (the synchronous case)}
\label{sec:analysis}

This section outlines the key ideas for the establishment of our main results of Q-learning for the synchronous case, namely Theorem~\ref{thm:infinite-horizon-simple} and Theorem~\ref{thm:LB-example}. The proof for TD learning is deferred to Appendix~\ref{sec:TD-learning-analysis}. Before delving into the proof details, we first introduce convenient vector and matrix notation that shall be used frequently. 

\input{matrix_notation.tex}

\input{proof_outline}

backgrounds.tex
\section{Background and algorithms}
\label{sec:background}

This paper concentrates on discounted infinite-horizon MDPs \citep{bertsekas2017dynamic}. We shall start by introducing some basics of tabular MDPs, followed by a description of both Q-learning and  TD learning. 
Throughout this paper,  
we denote by $\cS=\{1,\cdots, |\cS|\}$ and $\cA=\{1,\cdots, |\cA|\}$ the state space and the action space of the MDP, respectively, 
and let $\Delta(\cS)$  represent the probability simplex over the set $\cS$.  

%(resp. $\Delta(\cA)$)  (resp. $\cA$)

%\subsection{Q-learning for infinite-horizon discounted MDPs} 
%\label{sec:background-infinite}

\paragraph{Basics of discounted infinite-horizon MDPs.}

Consider an infinite-horizon MDP as represented by a quintuple $\mathcal{M} = (\cS,\cA, P, r,\gamma)$, where $\gamma\in (0,1)$ indicates the discount factor, 
 $P:\cS\times\cA \rightarrow \Delta(\cS)$ represents the probability transition kernel (i.e., $P(s' \mymid {s,a})$ is the probability of transiting to state $s'$ from a state-action pair $(s,a)\in \cS\times \cA$),  and $r:  \cS\times\cA \rightarrow [0,1]$ stands for the reward function (i.e., $r(s,a)$ is the immediate reward collected in state $s\in \cS$ when action $a\in \cA$ is taken). Note that the immediate rewards are assumed to lie within $[0,1]$ throughout this paper.  
Moreover, we let $\pi: \cS \rightarrow \Delta(\cA)$ represent a policy, so that $\pi( \cdot \mymid s)\in \Delta(\cA)$ specifies the (possibly randomized) action selection rule in state $s$. If $\pi$ is a deterministic policy, then we denote by $\pi(s)$ the action selected by $\pi$ in state $s$. 


 
%\paragraph{Value function and Q-function.}
A common objective in RL is to maximize a sort of long-term rewards called value functions or Q-functions. 
Specifically, given a policy $\pi$, the associated value function and Q-function of $\pi$ are defined respectively by
%
 \begin{align*}
   V^{\pi}(s) \defn \mathbb{E} \left[ \sum_{k=0}^{\infty} \gamma^k r(s_k,a_k ) \,\Big|\, s_0 =s \right]
\end{align*} 
%
 for all $s\in \cS$, and
 %
 \begin{equation*}
Q^{\pi}(s,a) \defn \mathbb{E} \left[ \sum_{k=0}^{\infty} \gamma^k r(s_k,a_k ) \,\Big|\, s_0 =s, a_0 = a \right]
\end{equation*} 
%
for all $(s,a)\in \cS \times \cA$. 
Here, $\{(s_k,a_k)\}_{k\geq 0}$ is a trajectory of the MDP induced by the policy $\pi$ (except $a_0$ when evaluating the Q-function), and the expectations are evaluated with respect to the randomness of  the MDP trajectory. 
Given that the immediate rewards fall within $[0,1]$, it can be straightforwardly verified that $0\leq V^{\pi}(s)\leq \frac{1}{1-\gamma} $ and $0\leq Q^{\pi}(s,a)\leq \frac{1}{1-\gamma} $ for any $\pi$ and any state-action pair $(s,a)$. The optimal  value function $V^\star$ and optimal Q-function $Q^\star$ are defined respectively as
%
\begin{equation*}
	V^{\star}(s) \coloneqq \max_{\pi} V^{\pi}(s), \qquad Q^{\star}(s,a) \coloneqq \max_{\pi} Q^{\pi}(s,a)
\end{equation*}
%
for any state-action pair $(s,a)\in \cS\times \cA$.  
It is well known that there exists a {\em deterministic} optimal policy, denoted by $\pi^{\star}$, that attains $V^{\star}(s)$ and $Q^{\star}(s,a)$ simultaneously for all $(s,a)\in \cS\times \cA$ \citep{sutton2018reinforcement}.    
 

\paragraph{Algorithms: Q-learning and TD learning (the synchronous setting).}

The synchronous setting assumes access to a generative model \citep{kearns1999finite,sidford2018near} such that:  
in each iteration $t$, we collect an independent sample $s_t(s,a) \sim P(\cdot \mymid s,a)$ for every state-action pair $(s,a)\in \cS\times \cA$. 


With this sampling model in place, the Q-learning algorithm \citep{watkins1992q} maintains a Q-function estimate $Q_t: \cS\times \cA \rightarrow \mathbb{R}$ for all $t\geq 0$; in each iteration $t$, the algorithm updates {\em all} entries of the Q-function estimate at once via the following update rule
%
\begin{equation}
\label{eqn:q-learning}
	Q_t  = (1- \eta_t ) Q_{t-1} + \eta_t \mathcal{T}_t (Q_{t-1}) . 
\end{equation}
%
Here, $\eta_t \in (0,1]$ denotes the learning rate or the step size in the $t$-th iteration,  
and $\mathcal{T}_t$ denotes the empirical Bellman operator constructed by samples collected in the $t$-th iteration, i.e., 
%
\begin{align}  \label{defn:empirical-Bellman-t-inf}
	\mathcal{T}_t (Q) (s , a ) &\coloneqq 
		 r(s , a ) + \gamma \max_{a' \in \cA} Q(s_t, a') , \qquad s_t  \equiv s_t(s,a) \sim P(\cdot \mymid s,a) 
\end{align}
%
for each state-action pair $(s,a)\in \cS\times \cA$. 
Obviously, $\mathcal{T}_t$ is an unbiased estimate of the celebrated Bellman operator $\mathcal{T}$ given by  
%
\begin{equation*}
	\forall (s,a)\in \cS\times \cA: \qquad
	\mathcal{T}(Q)(s,a) := r(s,a) + \gamma \mathop{\mathbb{E}}_{s^{\prime} \sim P(\cdot| s,a)}  \Big[ \max_{a^{\prime}\in \cA} Q(s^{\prime}, a^{\prime}) \Big] .
\end{equation*}
% 
Note that the optimal Q-function $Q^{\star}$ is the unique fixed point of the Bellman operator \citep{bellman1952theory}, that is, $\mathcal{T}(Q^{\star}) =Q^{\star}$.  Viewed in this light, synchronous Q-learning can be interpreted as a stochastic approximation scheme \citep{robbins1951stochastic} aimed at solving this fixed-point equation. 
Throughout this work, we initialize the algorithm in a way that obeys $0\leq Q_0(s,a) \leq \frac{1}{1-\gamma}$ for every state-action pair $(s,a)$.  
In addition, the corresponding value function estimate $V_t: \cS \rightarrow \mathbb{R}$ in the  $t$-th iteration is defined as
%
\begin{align}
	\label{defn:Vt}
	\forall s\in \cS: \qquad V_t(s) :=  \max_{a\in \cA} Q_{t}(s,a) .
\end{align}
%
The complete description of Q-learning is summarized in Algorithm~\ref{alg:q-infinite}.
 

 




\begin{algorithm}[t]
	\begin{algorithmic}[1] 
	\STATE\textbf{inputs:} learning rates $\{\eta_t\}$, number of iterations $T$, discount factor $\gamma$, initial estimate $Q_0$.
%\STATE  \textbf{initialization:} $Q_0=0$. 
   \FOR{$t=1,2,\cdots,T$}
	\STATE{ Draw  $s_t(s,a) \sim  P(\cdot \mymid s,a)$ for each $(s,a)\in \cS\times \cA$. }
	\STATE{ Compute $Q_t$ according to \eqref{eqn:q-learning} and \eqref{defn:empirical-Bellman-t-inf}.}
\ENDFOR
   %\textbf{return:} ${Q}_{T}$. 
   \end{algorithmic} 
    \caption{Synchronous Q-learning for infinite-horizon discounted MDPs.}
 \label{alg:q-infinite}
\end{algorithm}

 
As it turns out, TD learning \citep{sutton1988learning,tsitsiklis1997analysis,bhandari2021finite} in the synchronous setting can be viewed as a special instance of 
Q-learning when the action set $\cA$ is a singleton (i.e., $|\cA|=1$).    
In such a case, the MDP reduces to a Markov reward process (MRP) \citep{bertsekas2017dynamic}, and we shall abuse the notation to use  
 $P: \mathcal{S} \rightarrow \Delta(\cS)$ to describe the probability transition kernel, and employ $r: \mathcal{S} \rightarrow [0,1]$ to represent the reward function (with $r(s)$ indicating the immediate reward gained in state $s$). 
 The TD learning algorithm maintains an estimate $V_t: \mathcal{S} \rightarrow \mathbb{R}$ of the value function in each iteration $t$,\footnote{There is no need to maintain additional Q-estimates, as the Q-function and the value function coincide when $|\cA|=1$. } 
and carries out the following iterative update rule
 %
\begin{align}
\label{eqn:td-learning}
	V_t (s) &= (1- \eta_t ) V_{t-1} (s) + \eta_t \big( r(s) + \gamma   V_{t-1}(s_t) \big),   
	\qquad s_t \equiv s_t(s) \sim P(\cdot\mymid s)
\end{align}
%
for each state $s\in \cS$. 
As before, $\eta_t\in (0,1]$ is the learning rate at time $t$, the initial estimate $V_0(s)$ is taken to be within $\big[0, \frac{1}{1-\gamma}\big]$, and in each iteration, 
the samples $\{s_t(s) \mymid s\in \cS\}$ are generated independently.  
 The whole algorithm of TD learning is summarized in Algorithm~\ref{alg:td-infinite}. 
 

 \begin{algorithm}[t]
	\begin{algorithmic}[1] 
	\STATE\textbf{inputs:} learning rates $\{\eta_t\}$, number of iterations $T$, discount factor $\gamma$, initial estimate $V_0$. 
   \FOR{$t=1,2,\cdots,T$}
	\STATE{ Draw  $s_t(s) \sim  P(\cdot \mymid s)$ for each $s\in \cS$. }
	\STATE{ Compute $V_t$ according to \eqref{eqn:td-learning}.}
\ENDFOR
   %\textbf{return:} ${Q}_{T}$. 
   \end{algorithmic} 
    \caption{Synchronous TD learning for infinite-horizon discounted MRPs.}
 \label{alg:td-infinite}
\end{algorithm}




 
Finally, while synchronous Q-learning is the main focal point of this paper, we shall also discuss extension to the asynchronous Q-learning,  which we will elaborate on in Section~\ref{sec:AsynQ}. 

 


case3.tex
\subsubsection{Case 3: medium learning rates ($ 1-\gamma < \ceta < \log T$ or $ \frac{1}{(1-\gamma)T} \leq \eta \leq \frac{1}{(1-\gamma)^2T}$)}

Throughout this case, we assume that 
%
\begin{equation}
	\label{eq:eta0_lb_case3}
	\eta_{0}^{(T)}\leq \frac{1}{75}.
\end{equation}
%
In fact, if $\eta_{0}^{(T)}>1/75$, then the scenario becomes much easier to cope with. 
To see this, applying the previous result~\eqref{eqn:Vstar-lb-violin} and recalling the choice \eqref{defn-parameter-p} of $p$ immediately yield
%
\begin{align}
	V^{\star}(2) - \mathbb{E}\big[V_T(2)\big] \ge \frac{\eta_0^{(T)}}{1-\gamma p} > \frac{1}{100(1-\gamma)},
\end{align}
%
which together with \eqref{eq:bias-variance-decomposition} and the assumption $T\geq \frac{1}{(1-\gamma)^2}$ yields
%
\begin{align}
	\mathbb{E}\big[ \big( V^{\star}(2) - V_T(2) \big)^2 \big] 
	&\geq \big( V^{\star}(2) - \mathbb{E} \big[V_T(2)\big] \big)^2 
	%+ \mathsf{Var}\big(V_{T}(2)\big)  \notag\\
	 \geq \frac{1}{10000(1-\gamma)^2}
	 \geq \frac{1}{10000(1-\gamma)^4 T} .
	\label{eq:MSE-V2-case3-456}
\end{align}
%

We now turn our attention to the dynamics w.r.t.~state $1$
and its associated value function $V_{t}(1)$ under the condition \eqref{eq:eta0_lb_case3}. 


\paragraph{Two auxiliary sequences.}
Towards this, we first eliminate the effect of initialization on $Q_t(1, a)$ by introducing the following auxiliary sequence
%
\begin{align}
	\label{eqn:Qhat-schumann}
	\widehat{Q}_t(a) = (1-\eta_t)\widehat{Q}_{t-1}(a) + \eta_t\big\{ 1 + \gamma P_{t}(1\mymid 1, a)\widehat{V}_{t-1}\big\},
\end{align}
%
with 
%
\begin{align*}
	\widehat{V}_{t-1} \coloneqq \max_a \widehat{Q}_t(a) \qquad \text{and} \qquad \widehat{Q}_0(a) \coloneqq Q^{\star}(1, a) = \frac{1}{1-\gamma p},
\end{align*}
where we recall the value of $ Q^{\star}(1, a)$ from Lemma~\ref{lem:optimal-V-Q-hardMDP}.
In other words, $\{\widehat{Q}_t(a)\}$ is essentially a Q-learning sequence when initialized at the ground truth. 
Despite the difference in initialization, 
we claim that the discrepancy between $\widehat{Q}_t(a)$ and $Q_t(1, a)$ can be well controlled in the following sense:
%
\begin{align} 
\label{eq:Qt}
	Q_t(1, a) \ge \widehat{Q}_t(a) - \frac{1}{1-\gamma}\prod_{i = 1}^t \big(1-\eta_i(1-\gamma)\big) ,
	\qquad  a \in \{1,2\},
\end{align}
%
which shall be justified in Section~\ref{sec:proof-auxiliary-results-Case3}. As we shall discuss momentarily, the gap $\frac{1}{1-\gamma}\prod_{i = 1}^t \big(1-\eta_i(1-\gamma)\big)$ is sufficiently small for this case. 


Further, in order to control $\widehat{Q}_t(a)$, we find it convenient to introduce another auxiliary sequence as follows
%
\begin{align}
	\overline{Q}_t = (1-\eta_t)\overline{Q}_{t-1} + \eta_t\big\{ 1 + \gamma P_{t}(1\mymid 1, 1)\overline{Q}_{t-1}\big\} 
	\qquad \text{and} \qquad 
	\overline{Q}_0 = V^{\star}(1) = \frac{1}{1-\gamma p},
	\label{eq:sequence-overline-Qt}
\end{align}
%
which can be interpreted as a Q-learning sequence when there is only a single action (so that there is no max operator involved). 
In view of the basic fact that $\widehat{V}_t =\max_a \widehat{Q}_t(a) \ge \widehat{Q}_t(1)$, we can easily verify that
%
\begin{align}
	\widehat{Q}_t(1) \ge (1-\eta_t)\widehat{Q}_{t-1}(1) + \eta_t\big\{1 + \gamma P_{t}(1\mymid 1, 1)\widehat{Q}_{t-1}(1)\big\}
	\ge \overline{Q}_t,
	\label{eq:ordering-Qt-hat-overline}
\end{align}
%
allowing one to lower bound $\widehat{V}_t$ by controlling $\overline{Q}_t$. 


\paragraph{A useful lower bound on the auxiliary sequence~\eqref{eqn:Qhat-schumann}.}

In what follows, let us establish a useful lower bound on the sequence~\eqref{eqn:Qhat-schumann} introduced above. 
 Then we claim that there exists some $\tau \leq T$ (see \eqref{eq:defn-tau-case31} and \eqref{eq:defn-tau-case32}) such that  
%
\begin{align}
	\label{eqn:Tailbound-vhat-rolland}
	\mathbb{P}\Big\{\widehat{V}_t \ge \frac{1}{4(1-\gamma)} \Big\} \ge \frac{1}{2},\qquad \text{for}~t \ge \tau.
\end{align}
%
The auxiliary sequence constructed in \eqref{eq:sequence-overline-Qt} plays a crucial role in establishing this claim. 
 


\begin{proof}[Proof of the claim \eqref{eqn:Tailbound-vhat-rolland}]

We intend to employ the sequence $\overline{Q}_{t}$ (cf.~\eqref{eq:sequence-overline-Qt}) to help control $\widehat{V}_t$. 
It is first observed that the sequence $\overline{Q}_t$ admits the following decomposition (akin to the derivation in \eqref{eq:Vt2-expansion-case1})
%
\begin{align}
\notag \overline{Q}_{t} & =\big(1-\eta_{t}(1-\gamma p)\big)\overline{Q}_{t-1}+\eta_{t}\big\{1+\gamma\big(P_{t}(1\mymid1,1)-p\big)\overline{Q}_{t-1}\big\}\\
%
\notag & =\prod_{i=1}^{t}\big(1-\eta_{i}(1-\gamma p)\big)\overline{Q}_{0}+\sum_{k=1}^{t}\eta_{k}\prod_{i=k+1}^{t}\big(1-\eta_{i}(1-\gamma p)\big)\big\{1+\gamma\big(P_{k}(1\mymid1,1)-p\big)\overline{Q}_{k-1}\big\}\\
%
 \notag & =\eta_{0}^{(t)}  \frac{1}{1-\gamma p} +\sum_{k=1}^{t}\eta_{k}^{(t)}+\sum_{k=1}^{t}\eta_{k}^{(t)}\gamma\big(P_{k}(1\mymid1,1)-p\big)\overline{Q}_{k-1}\\
 & =\frac{1}{1-\gamma p}+\sum_{k=1}^{t}\underset{\eqqcolon\,z_{k}}{\underbrace{\eta_{k}^{(t)}\gamma\big(P_{k}(1\mymid1,1)-p\big)\overline{Q}_{k-1}}},
	\label{eqn:qbar-up-decomp}
\end{align}
%
where the last line results from \eqref{eq:eta-sum-hat}. In order to lower bound $\overline{Q}_{t}$, it boils down to controlling $\sum_k z_k$. 




Note that the sequence $\{z_k\}$ defined above is a martingale satisfying
%
\begin{align*}
	\mathbb{E}\big[ z_k\mymid &P_{k-1}(1\mymid 1, 1), \ldots, P_{1}(1\mymid 1, 1)\big] = 0 \\
	\text{and} &\qquad |z_k| \le \max_{1 \le k \le t} \eta_k^{(t)} \cdot \frac{\gamma p}{1-\gamma},
\end{align*}
%
where the last inequality follows from the basic property $0\leq \overline{Q}_{k-1}\leq \frac{1}{1-\gamma}$ (akin to Lemma~\ref{lemma:non-negativity-Qt-Vt}) and the fact that $\big| P_{k}(1\mymid1,1)-p\big|\leq \max \{p, 1-p\}=p$ since $p = (4\gamma-1)/(3\gamma)$ and $\gamma \geq 3/4$. We intend to invoke Freedman's inequality to control \eqref{eqn:qbar-up-decomp}. 
Armed with these properties and the fact that $P_k(1\mymid 1,1)$ is a Bernoulli random variable with mean $p$, we obtain
%
\begin{align*}
\sum_{k = 1}^t \mathsf{Var}\Big(z_k\mymid P_{k-1}(1\mymid 1, 1), \ldots, P_{1}(1\mymid 1, 1)\Big) &= \sum_{k = 1}^t \big( \eta_k^{(t)}\big)^2p(1-p)\big(\gamma\overline{Q}_{k-1}\big)^2 \\
&\le \max_{1 \le k \le t} \eta_k^{(t)} \cdot \sum_{k = 1}^t \eta_k^{(t)} \cdot \frac{1}{3(1-\gamma)} \le \frac{\max_{1 \le k \le t} \eta_k^{(t)}}{4(1-\gamma)^2}.
\end{align*}
%
Here, the penultimate inequality relies on the fact $0\leq \overline{Q}_{k-1}\leq \frac{1}{1-\gamma}$ (akin to Lemma~\ref{lemma:non-negativity-Qt-Vt}) and the choice of $p$ (see definition~\eqref{defn-parameter-p}), 
whereas the last inequality results from the following condition (derived through \eqref{eq:eta-sum-hat})
%
\[
	\sum_{k = 1}^t \eta_k^{(t)} = \big(1 - \eta_0^{(t)}\big)\frac{1}{1-\gamma p} \le \frac{1}{1-\gamma p} = \frac{3}{4(1-\gamma)} .
\]
%
Applying Freedman's inequality (see \eqref{eq:Freedman-special}) then yields 
%
\begin{align} 
	\label{eq:variance}
\mathbb{P}\left\{\bigg|\sum_{k = 1}^t z_k\bigg| \ge \sqrt{\frac{4\max_{1 \le k \le t} \eta_k^{(t)}}{(1-\gamma)^2}\log\frac{2}{\delta}} + \frac{4\max_{1 \le k \le t} \eta_k^{(t)}}{3(1-\gamma)}\log\frac{2}{\delta} \right\} \le \delta.
\end{align}
%



As an implication of the preceding result, a key ingredient towards bounding $\sum_{k = 1}^t z_k$ lies in controlling the quantity $\max_{1 \le k \le t} \eta_k^{(t)}$. To do so, we claim for the moment that there exists some $\tau\leq T$ such that 
 %
\begin{align}
	\label{eqn:schubert-winterreise}
	\max_{1 \le k \le t} \eta_k^{(t)} \le \frac{1}{50},\qquad\text{for}~t \ge \tau,
\end{align} 
%
whose proof is postponed to Section~\ref{sec:proof-auxiliary-results-Case3}. 
In light of this claim, setting $\delta = 1/2$ in the expression~\eqref{eq:variance} yields $$\sum_{k = 1}^t z_k \geq -\frac{1}{2(1-\gamma)}$$ with probably at least $1/2$. Combining this with the decomposition~\eqref{eqn:qbar-up-decomp} and  the property \eqref{eq:ordering-Qt-hat-overline}, we arrive at 
%
\begin{align*}
	\widehat{V}_t \geq \widehat{Q}_t(1) \geq \overline{Q}_t \geq \frac{1}{1-\gamma p} -\frac{1}{2(1-\gamma)}
	=  \frac{1}{4(1-\gamma)}
\end{align*}
%
with probability at least $1/2$, where the last identity relies on the choice of $p$ (see the definition~\eqref{defn-parameter-p}).
This establishes the advertised claim \eqref{eqn:Tailbound-vhat-rolland}. 
\end{proof}


%
\paragraph{Main proof.}
With the property \eqref{eqn:Tailbound-vhat-rolland} in place, we are positioned to prove our main result. 
Towards this, we find it convenient to define
%
\begin{subequations}
\begin{align}
	\Delta_t(a) &\defn \widehat{Q}_t(a) - Q^{\star}(1, a), \qquad a = 1, 2; \\
	\Delta_{t,\mathsf{max}} &\defn \max_a \Delta_t(a). 
\end{align}
\end{subequations}
%
The goal is thus to control $\Delta_{T,\mathsf{max}}$; in fact, we intend to show that $\Delta_{T,\mathsf{max}}$ is in expectation excessively large, resulting in an ``over-estimation'' issue that hinders convergence.
Towards this, it follows from the iterative update rule \eqref{eqn:Qhat-schumann} that 
%
\begin{align*}
\Delta_t(a) &= (1-\eta_t)\Delta_{t-1}(a) + \eta_t\big(1 + \gamma P_{t}(1\mymid 1, a)\widehat{V}_{t-1} - Q^{\star}(1, a)\big) \\
&= (1-\eta_t)\Delta_{t-1}(a) + \eta_t\gamma\big(P_{t}(1\mymid 1, a)\widehat{V}_{t-1} - pV^{\star}(1)\big) \\
	&= (1-\eta_t)\Delta_{t-1}(a) + \eta_t\gamma\big(p \big( \widehat{V}_{t-1} - V^{\star}(1) \big) + \big( P_{t}(1\mymid 1, a) - p \big) \widehat{V}_{t-1}\big) \\
&= (1-\eta_t)\Delta_{t-1}(a) + \eta_t\gamma\Big(p\Delta_{t-1,\mathsf{max}} + \big(P_{t}(1\mymid 1, a) - p \big)\widehat{V}_{t-1}\Big).
\end{align*}
%
Here, the second line comes from the Bellman equation $Q^{\star}(1, a) = 1 + \gamma pV^{\star}(1)$,
whereas the last line holds since $\widehat{V}_{t-1} - V^{\star}(1) = \max_a \big(\widehat{Q}_{t-1}(a) - V^{\star}(1)\big) = \max_a\Delta_{t-1}(a)$ (as a consequence of the relation~\eqref{eq:optimal-V-Q-hardMDP}). 
Applying the above relation recursively leads to
%
\begin{align}
\Delta_t(a) = \sum_{k = 1}^t\eta_k\prod_{i = k+1}^t \big(1-\eta_i\big)\gamma\Big(p\Delta_{k-1,\mathsf{max}} + \big(P_{k}(1\mymid 1, a) - p \big)\widehat{V}_{k-1}\Big),
\end{align}
%
where we have used the initialization $\Delta_0(a) = 0$. Letting 
%
\begin{subequations}
	\label{eqn:defn-xi-t}
\begin{align}
	\xi_t(a) &\defn \sum_{k = 1}^t\eta_k\prod_{i = k+1}^t \big(1-\eta_i\big)\gamma \big( P_{k}(1\mymid 1, a) - p \big)\widehat{V}_{k-1}, \\
	\xi_{t,\mathsf{max}} &\defn \max_a \xi_t(a),
\end{align}
\end{subequations}
%
%Setting $\Delta_t \defn \max_a \Delta_t(a)$, 
one can express the above relation as follows 
%
\begin{align*}
\Delta_{t,\mathsf{max}} = \sum_{k = 1}^t\eta_k\prod_{i = k+1}^t \big(1-\eta_i\big)\gamma p \Delta_{k-1,\mathsf{max}} + \xi_{t,\mathsf{max}}.
\end{align*}
%

Next, we claim that $\mathbb{E} [ \xi_{t,\mathsf{max}} ]$ satisfies the following property 
%
\begin{align}
\label{eqn:lb-xi}
	\mathbb{E} [ \xi_{t,\mathsf{max}} ] &\ge \frac{c}{\sqrt{(1-\gamma)^2T}\log T}
	\qquad \text{for all}~t \ge \widehat{\tau}
\end{align}
%
for some universal constant $c>0$,  where
%
\begin{align}
\label{eq:tau-hat-byproduct-duplicate}
\widehat{\tau} \defn \max\Bigg\{\tau' \,\,\Big|\,\, \prod_{i = \tau'}^{T} \big(1-\eta_i(1-\gamma p)\big) \le \frac{6}{7}\Bigg\}	,
\end{align}
%
whose existence is ensured under the condition \eqref{eq:eta0_lb_case3}.
Given the validity of this claim (which we shall justify in Section~\ref{sec:proof-auxiliary-results-Case3}), we immediately arrive at
%
\begin{align}
\label{eqn:Delta-iterates-exp}
	\mathbb{E} [ \Delta_{t,\mathsf{max}} ] &\ge \sum_{k = 1}^t\eta_k\prod_{i = k+1}^t \big(1-\eta_i\big)\gamma p \mathbb{E} [ \Delta_{k-1,\mathsf{max}} ] + \frac{c}{\sqrt{(1-\gamma)^2T}\log T}\qquad \text{for all}~t \ge \widehat{\tau}.
\end{align}
%


In order to study the above recursion, it is helpful to look at the following sequence
%
\begin{align}
\label{eqn:new-sequence}
x_t = (1-\eta_t)x_{t-1} + \eta_t \bigg(\gamma p x_{t-1} + \frac{c}{\sqrt{(1-\gamma)^2T}\log T} \bigg)
\end{align}
%
with $x_{\widehat{\tau}} = 0$, where we recall the definition of $\widehat{\tau}$ in \eqref{eq:tau-hat-byproduct-duplicate}.  
In comparison to the iterative relation~\eqref{eqn:Delta-iterates-exp} which starts from $\mathbb{E} [ \Delta_{0,\mathsf{max}} ] = 0$ (and hence $\mathbb{E} [ \Delta_{t,\mathsf{max}} ] \geq 0$), we let the sequence $x_t$ start from $x_{\widehat{\tau}} = 0$, where $\widehat{\tau}$ is defined in \eqref{eq:tau-hat-byproduct-duplicate}.
It is straightforward to verify that 
%
\begin{equation}
	\mathbb{E} \big[ \Delta_{T,\mathsf{max}} \big] \ge x_T,
	\label{eq:E-Delta-Tmax-xT-order}
\end{equation}
%
recognizing that
%
\begin{align*}
x_t = \sum_{k = \widehat{\tau}}^t\eta_k\prod_{i = k+1}^t \big(1-\eta_i\big)\gamma p x_{k-1} + 
 \sum_{k = \widehat{\tau}}^T \eta_k\prod_{i = k+1}^T
\big(1- \eta_i)\frac{c}{\sqrt{(1-\gamma)^2T}\log T}. 
\end{align*}
%
A little algebra reveals that the sequence~\eqref{eqn:new-sequence} obeys
%
\begin{align*}
	x_T &= \frac{c}{\sqrt{(1-\gamma)^2T}\log T} \sum_{k = \widehat{\tau}}^T \eta_k\prod_{i = k+1}^T
\big(1-\eta_i(1-\gamma p) \big) 
= \frac{c}{\sqrt{(1-\gamma)^2T}\log T}\frac{1}{1-\gamma p}\left[1-\prod_{i = \widehat{\tau}}^T
\big(1-\eta_i(1-\gamma p) \big)\right] \\
&= \frac{3c}{4\sqrt{(1-\gamma)^4 T}\log T}\left[1-\prod_{i = \widehat{\tau}}^T
\big(1-\eta_i(1-\gamma p) \big)\right]
\ge \frac{3c}{28\sqrt{(1-\gamma)^4T}\log T},
\end{align*}
%
where the second equality arises from \eqref{eq:eta-sum-hat-tau},
and the last inequality holds as long as $\prod_{i = \widehat{\tau}}^T \big(1-\eta_i(1-\gamma p)\big) \le 6/7$ (see \eqref{eq:tau-hat-byproduct}). 
This taken together with \eqref{eq:E-Delta-Tmax-xT-order} leads to 
%
\begin{align*}
	\mathbb{E} \big[ \Delta_{T,\mathsf{max}} \big] \ge x_T \ge \frac{3c}{28\sqrt{(1-\gamma)^4T}\log T} .
\end{align*}
%



Combining the above bound with \eqref{eq:Qt} leads to
%
\begin{align*} 
	\mathbb{E} \Big[ V_T(1) - V^{\star}(1) \Big] &\ge \mathbb{E} \Big[  \Delta_{T,\mathsf{max}} - \frac{1}{1-\gamma}\prod_{i = 1}^T \big(1-\eta_i(1-\gamma)\big) \Big] \\
	&\ge \frac{3c}{28\sqrt{(1-\gamma)^4T}\log T} - \frac{1}{1-\gamma}\prod_{i = 1}^T \big(1-\eta_i(1-\gamma)\big).
\end{align*}
%
Taking this together with \eqref{eq:Vt3-expansion-case3}, we arrive at
%
\begin{align*} 
	%\max \mathbb{E} \Big[ \big|V_T(s) - V^{\star}(s)\big| \Big] 
\max \Big\{	&\mathbb{E} \Big[ \big|V_T(3) - V^{\star}(3) \big| \Big],\;  \mathbb{E} \Big[ \big|V_T(1) - V^{\star}(1) \big| \Big] \Big\} \\
	&\ge \max\left\{\frac{1}{1-\gamma}\prod_{i = 1}^T \big(1-\eta_i(1-\gamma)\big), ~\frac{3c}{28\sqrt{(1-\gamma)^4T}\log T} - \frac{1}{1-\gamma}\prod_{i = 1}^T \big(1-\eta_i(1-\gamma)\big) \right\} \\
	&\ge \frac{1}{2} \cdot \frac{1}{1-\gamma}\prod_{i = 1}^T \big(1-\eta_i(1-\gamma)\big) + \frac{1}{2}  \left[\frac{3c}{28\sqrt{(1-\gamma)^4T}\log T} - \frac{1}{1-\gamma}\prod_{i = 1}^T \big(1-\eta_i(1-\gamma)\big)\right] \\
	&= \frac{3c}{56\sqrt{(1-\gamma)^4T}\log T}.
\end{align*}
%
This combined with \eqref{eq:bias-variance-decomposition}  establishes the following desired lower bound: 
%
\begin{align*} 
	\max_s \mathbb{E} \Big[ \big|V_T(s) - V^{\star}(s)\big|^2 \Big] 
	\geq \left( \frac{3c}{56\sqrt{(1-\gamma)^4T}\log T} \right)^2 = \frac{9c^2}{56^2 (1-\gamma)^4T \log^2 T}.
\end{align*}
%

 



\subsubsection{Proofs of auxiliary results}
\label{sec:proof-auxiliary-results-Case3}

\paragraph{Proof of the inequality \eqref{eq:Qt}.} 

We shall establish this claim by induction. 
To begin with, the inequality \eqref{eq:Qt} holds trivially for the base case with $t=0$.
Now, let us assume that the claim holds up to the $(t-1)$-th iteration, and we would like to justify it for the $t$-th iteration.
As an immediate consequence of the claim \eqref{eq:Qt} for the $(t-1)$-th iteration and the definitions of $V_{t-1}$ and  $\widehat{V}_{t-1}$, we have 
%
\begin{align*} 
	V_{t-1}(1) = \max_{a} Q_{t-1}(1, a) 
	&\ge \max_{a}  \widehat{Q}_{t-1}(a) - \frac{1}{1-\gamma}\prod_{i = 1}^{t-1} \big(1-\eta_i(1-\gamma)\big) \\
	&= \widehat{V}_{t-1} - \frac{1}{1-\gamma}\prod_{i = 1}^{t-1} \big(1-\eta_i(1-\gamma)\big) .
\end{align*}
%
By virtue of the respective update rules of $Q_t(1, a)$ and $\widehat{Q}_t(a)$, we can express their difference as follows:
%
\begin{align*}
Q_t(1, a) - \widehat{Q}_t(a) &= (1-\eta_t)\big(Q_{t-1}(1, a) - \widehat{Q}_{t-1}(a)\big) + \eta_t\gamma P_{t}(1\mymid 1, a)\big(V_{t-1}(1) - \widehat{V}_{t-1}\big) \\
&\ge -(1-\eta_t)\frac{1}{1-\gamma}\prod_{i = 1}^{t-1} \big(1-\eta_i(1-\gamma)\big) - \eta_t\gamma P_{t}(1\mymid 1, a) \frac{1}{1-\gamma}\prod_{i = 1}^{t-1} \big(1-\eta_i(1-\gamma)\big) \\
&\ge -(1-\eta_t)\frac{1}{1-\gamma}\prod_{i = 1}^{t-1} \big(1-\eta_i(1-\gamma)\big) - \eta_t\gamma  \frac{1}{1-\gamma}\prod_{i = 1}^{t-1} \big(1-\eta_i(1-\gamma)\big) \\
&= -\frac{1}{1-\gamma}\prod_{i = 1}^t \big(1-\eta_i(1-\gamma)\big), 
\end{align*}
%
where the first inequality invokes the induction hypothesis for the $(t-1)$-th iteration. 
This establishes \eqref{eq:Qt} for the $t$-th iteration, and hence the proof is complete via an induction argument.
%

 
\paragraph{Proof of the claim~\eqref{eqn:schubert-winterreise}.}
When taking the constant learning rates $\eta_t \equiv \eta \le \frac{1}{(1-\gamma)^2T} \le \frac{1}{50}$ (under the condition $T \ge \frac{50}{(1-\gamma)^2}$), one has
%
\begin{align*}
\max_{1 \le k \le t} \eta_k^{(t)} \le \eta_t =\eta \le \frac{1}{50},
\end{align*} 
%
thus allowing us to take $\tau=1$ for this case.



It then suffices to look at rescaled linear learning rates (i.e., $\eta_t = \frac{1}{1+\ceta(1-\gamma)t}$). 
As already calculated in the expression~\eqref{eq:ratio-etakT-eta-kminus1}, the ratio of two consecutive quantities obeys 
%
\begin{align}
\label{eqn:ratio-schubert}
\frac{\eta_{k-1}^{(t)}}{\eta_k^{(t)}} 
%=\frac{\eta_{k-1}\prod_{i = k}^t \big(1-\eta_i(1-\gamma p)\big)}
%{\eta_{k}\prod_{i = k+1}^t \big(1-\eta_i(1-\gamma p)\big)}
= \frac{1-\frac{4}{3}(1-\gamma)\eta_k}{1-c_\eta(1-\gamma)\eta_k}.
\end{align}
%
% which means that $\eta_k^{(t)}$ is monotonic in $t$.
In what follows, we divide into two cases, depending on whether this sequence is decreasing or increasing.


\begin{itemize}

		
\item {\em The case with $4/3 \le \ceta < \log T$.} 
In this scenario, the ratio~in \eqref{eqn:ratio-schubert} is larger than 1, and hence the sequence $\{\eta_{k}^{(t)}\}$ decreases with $k$. 
Let us define 
%
\begin{align}
	\tau \defn \min\Bigg\{\tau' \,\,\Big|\,\, \prod_{i = 1}^{\tau'} \big(1-\eta_i(1-\gamma p)\big) \le \frac{1}{50}\Bigg\},	
	\label{eq:defn-tau-case31}
\end{align}
%
which clearly satisfies $\tau \leq T$ (in view of \eqref{eq:eta0_lb_case3}). 
For all $t \ge \tau$, one has
%
\begin{align*}
\max_{1 \le k \le t} \eta_k^{(t)} \le \prod_{i = 1}^{\tau} \big(1-\eta_i(1-\gamma p)\big) \le \frac{1}{50}.
\end{align*} 
%
At the same time, we claim that one must have 
%
\begin{align}
	\prod_{i = \tau}^{T} \big(1-\eta_i(1-\gamma p)\big) \le \frac{2}{3}.
	\label{eq:prod-etai-tau-T-23}
\end{align}
%
Otherwise, recalling $\eta_0^{(T)} = \prod_{i = 1}^{T} \big(1-\eta_i(1-\gamma p)\big)$, we have 
%
$$
\eta_0^{(T)} = \left\{ \prod_{i = 1}^{\tau-1} \big(1-\eta_i(1-\gamma p)\big) \right\}
\left\{ \prod_{i = \tau}^{T} \big(1-\eta_i(1-\gamma p)\big)   \right\} > \frac{1}{50} \cdot \frac{2}{3}=\frac{1}{75}, 
$$ 
%
which contradicts our assumption that $\eta_0^{(T)} > 1/75 $ (cf.~\eqref{eq:eta0_lb_case3}). 

 

\item {\em The case with $1-\gamma < \ceta < 4/3$.}
In this case, the sequence $\eta_{k}^{(t)}$ increases with $k$. 
If we set  
%
\begin{align}
	\tau \defn \Big\lceil\frac{49}{\ceta(1-\gamma)} \Big\rceil
	< \frac{50}{(1-\gamma)^2}< T,
	\label{eq:defn-tau-case32}
\end{align}
%
then for all $t \ge \tau$ we have 
%
\begin{align*}
\max_{1 \le k \le t} \eta_k^{(t)} = \eta_t^{(t)} = \eta_t \le \eta_{\tau} 
\leq 
\frac{1}{1+\ceta(1-\gamma)\frac{49}{\ceta(1-\gamma)}}
= \frac{1}{50}. 
\end{align*} 
%
Under the condition $T \ge \frac{150}{(1-\gamma)^2}\geq \frac{150}{\ceta(1-\gamma)}$ (so that $T-\tau +1 \geq \frac{100}{\ceta(1-\gamma)} \geq \frac{100}{(1-\gamma)4/3}$), 
one can show that
%
\begin{align}
	\prod_{i = \tau}^{T} \big(1-\eta_i(1-\gamma p)\big) \le \Big( 1-\frac{1-\gamma}{100} \Big)^{T-\tau+1}
\leq \Big( 1-\frac{1-\gamma}{100} \Big)^{\frac{100}{(1-\gamma)4/3}} \le \frac{3}{4}.
	\label{eq:prod-etai-tau-T-34}
\end{align}
 


\item Putting these two cases together (with $\tau$ specified in \eqref{eq:defn-tau-case31} and \eqref{eq:defn-tau-case32}), we obtain  
%
\begin{align} 
\max_{1 \le k \le t} \eta_k^{(t)} \le \frac{1}{50}
\end{align}
%
for all $t \ge \tau$, thus establishing the desired inequality~\eqref{eqn:schubert-winterreise}.


\end{itemize}

 


 

\paragraph{Proof of the inequality~\eqref{eqn:lb-xi}.} 
%
For every $t$, recalling the definition~\eqref{eqn:defn-xi-t}, it is convenient to write 
%
\begin{align*}
	\mathbb{E} [\xi_{t,\mathsf{max}}] &= \mathbb{E} \left[ \frac{\xi_t(1)+\xi_t(2)+|\xi_t(1)-\xi_t(2)|}{2} \right] 
	= \mathbb{E} \left[ \frac{|\xi_t(1)-\xi_t(2)|}{2}  \right] \\
	&= \frac{1}{2}\mathbb{E} \left[ \, \left|\sum_{k = 1}^t\eta_k\prod_{i = k+1}^t \big(1-\eta_i\big)\gamma \big( P_{k}(1\mymid 1, 1) -  P_{k}(1\mymid 1, 2) \big)\widehat{V}_{k-1}) \,\right| \right] ,
\end{align*}
%
where we have used the fact that $\mathbb{E}[\xi_t(a)] = 0$. 
To control the right-hand side of the above equation, let us define  
%
\begin{align*}
	\zeta_t &\defn \sum_{k = 1}^t z_k ,
	\qquad z_k \defn \eta_k\prod_{i = k+1}^t \big(1-\eta_i\big)\gamma \big(P_{k}(1\mymid 1, 1) - P_{k}(1\mymid 1, 2) \big)\widehat{V}_{k-1}
\end{align*}
%
for any $k\geq 1$, 
where $\{z_k\}$ also forms a martingale sequence since
%
\begin{align*}
	\mathbb{E}\Big[ z_k\mymid \big\{ P_{j}(1\mymid 1, 1), P_{j}(1\mymid 1, 2) \big\}_{1\leq j< k} \Big] = 0.
\end{align*}
%


As a consequence of Freedman's inequality, we claim that $\zeta_t $ satisfies
%
\begin{align}
\label{eqn:tail-bunny}
\mathbb{P}\left\{\big|\zeta_t\big| \ge \sqrt{\frac{8\log\frac{2}{\delta}}{3(1-\gamma)}\sum_{k = 1}^t\eta_k^2\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2} + \frac{4\eta_t \log\frac{2}{\delta}}{3(1-\gamma)} \right\} \le \delta. 
\end{align}
%
To verify this relation, we first notice that 
%
\begin{align}
\label{eqn:unif-bound}
|z_k| \le \max_{1 \le k \le t} \eta_k\prod_{i = k+1}^t \big(1-\eta_i\big) \cdot \frac{1}{1-\gamma} \le \frac{\eta_t}{1-\gamma} ,
\end{align}
%
provided that $\max_{k} \eta_k\prod_{i = k+1}^t \big(1-\eta_i\big) \le \eta_t$.
To verify the condition $\max_{k} \eta_k\prod_{i = k+1}^t \big(1-\eta_i\big) \le \eta_t$,  one can check --- similar to \eqref{eq:ratio-etakT-eta-kminus1} --- that
%
\begin{equation}\label{eq:ratio-check_eta_000}
\frac{\eta_{k-1}\prod_{i = k}^t \big(1-\eta_i\big)}{\eta_k\prod_{i = k+1}^t \big(1-\eta_i\big)} = 1 - \frac{\big(1 - \ceta(1-\gamma)\big)\eta_k}{1 - \ceta(1-\gamma)\eta_k}
	\leq 1,
\end{equation}
%
which indicates that $\eta_k\prod_{i = k+1}^t \big(1-\eta_i\big)$ is an increasing sequence as long as $\ceta \le \log T \le \frac{1}{1-\gamma}$ (see \eqref{eq:logT_ub}).  
In addition to the boundedness condition \eqref{eqn:unif-bound}, we can further calculate 
%
\begin{align*}
	& \sum_{k = 1}^t \mathsf{Var}\big(z_k\mymid P_{k-1}(1\mymid 1, 1), P_{k-1}(1\mymid 1, 2), \ldots, P_{1}(1\mymid 1, 1), P_{1}(1\mymid 1, 2)\big) \\
	&\qquad = \sum_{k = 1}^t\eta_k^2\left[\prod_{i = k+1}^t \big(1-\eta_i\big)\right]^2\cdot2p(1-p)\cdot\big(\gamma\widehat{V}_{k-1}\big)^2   \le \sum_{k = 1}^t\eta_k^2\left[\prod_{i = k+1}^t \big(1-\eta_i\big)\right]^2\cdot \frac{2}{3(1-\gamma)}, 
\end{align*}
%
where the last inequality comes from the facts that $\widehat{V}_{k-1} \le \frac{1}{1-\gamma}$ and the choice $p = \frac{4\gamma - 1}{3\gamma}$.
These bounds taken together with Freedman's inequality (see \eqref{eq:Freedman-special}) validate \eqref{eqn:tail-bunny}. 
 

By virtue of \eqref{eqn:tail-bunny}, setting $\delta = \frac{(1-\gamma)^2}{2}\mathbb{E} \big[ |\zeta_t|^2 \big]$ yields that with probability at least $1-\delta$, 
%
\begin{align} \label{eq:B-delta}
|\zeta_t| \le B	\defn 
\sqrt{\frac{8\log\frac{2}{\delta}}{3(1-\gamma)} \sum_{k = 1}^t\eta_k^2\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2} + \frac{4\eta_t \log\frac{2}{\delta}}{3(1-\gamma)} ~~
\text{ with } \delta = \frac{(1-\gamma)^2}{2}\mathbb{E} \big[ |\zeta_t|^2 \big]. 
\end{align}
%
When $T \ge \frac{1}{(1-\gamma)^2}$, one can ensure that 
%
\begin{align}
	\mathbb{E} [\xi_{t,\mathsf{max}}]  
	&= \frac{1}{2}\mathbb{E} \big[ |\zeta_t| \big]
	\ge \frac{1}{2}\mathbb{E} \big[ |\zeta_t|\mathds{1}\big(|\zeta_t| \le B\big) \big]  
	\ge \frac{1}{2B}\mathbb{E} \big[ |\zeta_t|^2\mathds{1}\big(|\zeta_t| \le B\big) \big] \nonumber \\
	&= \frac{1}{2B}\Big\{ \mathbb{E} [ |\zeta_t|^2 ] - \mathbb{E} \big[ |\zeta_t|^2\mathds{1}\big(|\zeta_t| > B\big) \big] \Big\} \nonumber \\
	&\overset{(\mathrm{i})}{\geq} \frac{1}{2B}\Big\{ \mathbb{E} [ |\zeta_t|^2 ] - \frac{1}{(1-\gamma)^2} \mathbb{P}\big\{|\zeta_t| > B\big\} \Big\} \nonumber \\
	&\ge \frac{1}{2B}\Big\{ \mathbb{E} [ |\zeta_t|^2 ] - \frac{\delta}{(1-\gamma)^2} \Big\} 
	\overset{(\mathrm{ii})}{\geq} \frac{1}{4B}\mathbb{E} [ |\zeta_t|^2 ]. \label{eq:emax_intermediate_lb}
	% &\stackrel{(i)}{\ge} \frac{c}{\sqrt{(1-\gamma)^2T}\log T},
\end{align}
%
Here, (i) holds since 
%
\begin{align*}
\big|\zeta_t\big| \le  \sum_{k=1}^t |z_k| \leq  \left[ \sum_{k=1}^t \eta_k\prod_{i = k+1}^t \big(1-\eta_i\big) \right] \cdot \frac{1}{1-\gamma} \leq \frac{1}{1-\gamma}
\end{align*} 
%
as a consequence of \eqref{eqn:unif-bound} and \eqref{eq:sum-eta-i-t-infinite}; (ii) holds by the choice of $\delta$.
It is thus sufficient to lower bound $\mathbb{E} [|\zeta_t|^2]$.
Towards this,  let us  define
%
\begin{align}
\label{eq:tau-hat-byproduct}
\widehat{\tau} \defn \max\Bigg\{\tau' \,\,\Big|\,\, \prod_{i = \tau'}^{T} \big(1-\eta_i(1-\gamma p)\big) \le \frac{6}{7}\Bigg\}	,
\end{align}
%
which clearly satisfies $\tau\leq \widehat{\tau}\leq T$ (in view of \eqref{eq:prod-etai-tau-T-23} and \eqref{eq:prod-etai-tau-T-34}).
%
Then, for all $t \ge \widehat{\tau}$ one has 
(which shall be proved towards the end of this subsection)
%
\begin{align}
	\sum_{k = \tau}^{t} \eta_k^2 \Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big] ^2
	\ge \frac{1}{8}\sum_{k = 1}^{t} \eta_k^2 \Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2.
	\label{eq:sum-LB-1/8-4567}
\end{align}

We now proceed to lower bound $\mathbb{E} [|\zeta_t|^2]$ for $t \ge \widehat{\tau}$. We first observe that for any $t \ge \widehat{\tau}$, 
%
\begin{align*}
	\mathbb{E} \big[ |\zeta_t|^2 \big] 
	&\ge \sum_{k = 1}^t \mathbb{E} \left[ \mathsf{Var}\Big(\eta_k\prod_{i = k+1}^t \big(1-\eta_i\big)\gamma\big(P_{k}(1\mymid 1, 1) - P_{k}(1\mymid 1, 2) \big)\widehat{V}_{k-1} \mymid \widehat{V}_{k-1}\Big) \right] \\
	&\ge  \frac{1}{2}\sum_{k = \tau}^t \mathbb{E} \left[ \mathsf{Var}\Big(\eta_k\prod_{i = k+1}^t \big(1-\eta_i\big)\gamma \big(P_{k}(1\mymid 1, 1) - P_{k}(1\mymid 1, 2) \big)\widehat{V}_{k-1} \mymid \widehat{V}_{k-1} \ge \frac{1}{4(1-\gamma)} \Big) \right],
\end{align*}
%
where the first line relies on \eqref{eq:variance-expansion-martingale}, 
and the last step makes use of the fact \eqref{eqn:Tailbound-vhat-rolland}. 
To further control the right-hand side of the above inequality, we take $\tau' \defn \max\left\{t - \frac{1}{\eta_{t/2}}, 1\right\} $  and show that 
%
\begin{align}
	\mathbb{E} \big[ |\zeta_t|^2 \big]  
	&\overset{\text{(i)}}{\ge} \frac{1}{2}\sum_{k = \tau}^t\eta_k^2\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2 \gamma^2\cdot 2p(1-p) \frac{1}{16(1-\gamma)^2}
	\nonumber\\
	&\ge \frac{1}{48(1-\gamma)}\sum_{k = \tau}^t\eta_k^2\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2 \overset{\text{(ii)}}{\ge} \frac{1}{400(1-\gamma)}\sum_{k = 1}^t\eta_k^2\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2 \nonumber\\
&\ge \frac{1}{400(1-\gamma)}
	\sum_{k = \tau'}^t\eta_k^2\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2 \overset{\text{(iii)}}{\ge} \frac{\eta_t}{9600(1-\gamma)}. \label{eq:square-zeta}
\end{align}
%
Here, (i) makes use of the constraint $\widehat{V}_{k-1} \ge \frac{1}{4(1-\gamma)}$, 
while (ii) makes use of \eqref{eq:sum-LB-1/8-4567}, and (iii) are valid if the following property holds (which shall be proved towards the end of this subsection)
%
%\begin{align}
%	\sum_{k = \tau}^{t} \eta_k^2 \Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big] ^2
%	\ge \frac{1}{8}\sum_{k = 1}^{t} \eta_k^2 \Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2;
%	\label{eq:sum-LB-1/8-4567}
%\end{align}
%
\begin{align}
	\sum_{k = \tau'}^t\eta_k^2\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2
	\ge \frac{1}{24} \eta_t.
	\label{eq:sum-LB-1/8-890}
\end{align}
%



We are now well-equipped to control $\mathbb{E} [\xi_{t,\mathsf{max}}] $
using the property \eqref{eq:emax_intermediate_lb}. Recall the expression of $B$ in \eqref{eq:B-delta}, we know that bounding $\mathbb{E} [ |\zeta_t|^2 ] / B$ boils down to  controlling
%
\begin{align}
\frac{\mathbb{E} [ |\zeta_t|^2 ]}{\frac{\eta_t}{1-\gamma}\log\frac{2}{\delta}} \qquad \mbox{and}\qquad \frac{\mathbb{E} [ |\zeta_t|^2 ]}{\sqrt{\frac{\log\frac{2}{\delta}}{1-\gamma}\sum_{k = 1}^t\eta_k^2\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2}} . 
	\label{eq:two-terms-LB-123}
\end{align}
%
\begin{itemize}
\item
For the first term in \eqref{eq:two-terms-LB-123}, recalling that $\delta = \frac{(1-\gamma)^2}{2}\mathbb{E} \big[ |\zeta_t|^2 \big]$, we can demonstrate that
%
\begin{align}
\log \frac{1}{\delta} = -\log \frac{(1-\gamma)^2}{2}\mathbb{E} \big[ |\zeta_t|^2 \big] \le -\log \frac{(1-\gamma)\eta_t}{19200} \le \log \frac{19200(1+(1-\gamma)T\log T)}{1-\gamma} \lesssim \log T ,
	\label{eq:log-delta-T-relation-123}
\end{align}
%
where the first inequality makes use of the bound \eqref{eq:square-zeta}, and the second inequality arises from the fact $\eta_t \geq \frac{1}{1+(1-\gamma)T\log T}$ (given the range of the learning rates in this case). 
Combining this with \eqref{eq:square-zeta}, 
we can guarantee that 
%
\[
\frac{\mathbb{E} [ |\zeta_t|^2 ]}{\frac{\eta_t}{1-\gamma}\log\frac{2}{\delta}} \gtrsim \frac{1}{\log T}. 
\]
%

\item
Moving to the second term in \eqref{eq:two-terms-LB-123}, one can ensure that
%
\begin{align*}
\frac{\mathbb{E} \big[ |\zeta_t|^2 \big]}{\sqrt{\frac{\log\frac{2}{\delta}}{1-\gamma}\sum_{k = 1}^t\eta_k^2\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2}} & \overset{\mathrm{(i)}}{\gtrsim} \sqrt{\frac{1}{(1-\gamma)\log T}\sum_{k = 1}^t\eta_k^2\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2} \\
& \overset{\mathrm{(ii)}}{\gtrsim}  \sqrt{\frac{\eta_t}{(1-\gamma)\log T}} \overset{\mathrm{(iii)}}{\gtrsim} \frac{1}{\sqrt{(1-\gamma)^2T}\log T}.
\end{align*}
%
Here, (i) follows from \eqref{eq:square-zeta} and \eqref{eq:log-delta-T-relation-123} since
%
\[
	\mathbb{E} \big[ |\zeta_t|^2 \big] \gtrsim \frac{1}{1-\gamma}\sum_{k = 1}^t\eta_k^2\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2\qquad \text{and} \qquad \log\frac{2}{\delta} \lesssim \log T;
\]
(ii) arises from \eqref{eq:sum-LB-1/8-890}; and (iii) relies on the fact $\eta_t \gtrsim \frac{1}{(1-\gamma)T\log T}$ (given the range of the learning rates in this case).
%
\end{itemize}

Substituting the above relations into  \eqref{eq:emax_intermediate_lb} and using the expression of $B$ in \eqref{eq:B-delta}, we reach at
%
\begin{align*}
	\mathbb{E} [ \xi_{t, \mathsf{max}} ] \ge \frac{1}{4B}\mathbb{E} \big[ |\zeta_t|^2 \big]
	&\ge \frac{c}{\sqrt{(1-\gamma)^2T}\log T},
\end{align*}
%
for some constant $c > 0$. Thus, this validates the inequality~\eqref{eqn:lb-xi}. 


\begin{proof}[Proof of the claim \eqref{eq:sum-LB-1/8-4567}] By the definition of $\widehat{\tau}$ in \eqref{eq:tau-hat-byproduct}, we have 
$\prod_{i = \widehat{\tau}}^{T} \big(1-\eta_i(1-\gamma p)\big) \le 6/7$. An important observation is that
\begin{align}\label{eqn:death-maiden}
\sum_{k = \tau}^{t} \eta_k\left[\prod_{i = k+1}^t \big(1-\eta_i\big)\right] 
	\stackrel{\mathrm{(i)}}{=} 1-\prod_{i = \tau}^t \big(1-\eta_i\big) 
	\stackrel{\mathrm{(ii)}}{\ge} \frac{1}{8}
	\stackrel{\mathrm{(iii)}}{\geq} \frac{1}{8}\sum_{k = 1}^{t} \eta_k\left[\prod_{i = k+1}^t \big(1-\eta_i\big)\right]. 
\end{align}
%
Here, the relations (i) and (iii) arise from \eqref{eq:sum-eta-i-t-infinite-tau}, and the inequality~(ii) follows since 
%
\begin{align}
\prod_{i = \tau}^t \big(1-\eta_i\big) \le \prod_{i = \tau}^{\widehat{\tau}} \big(1-\eta_i\big) \le \prod_{i = \tau}^{\widehat{\tau}} \big(1-\eta_i(1-\gamma p)\big) = \frac{\prod_{i = \tau}^{T} \big(1-\eta_i(1-\gamma p)\big)}{\prod_{i = \widehat{\tau}+1}^{T} \big(1-\eta_i(1-\gamma p)\big)} 
\leq \frac{\, 3/4\,}{6/7} \le \frac{7}{8},
\label{eq:prod-etai-tau-T-2345}
\end{align}
%
where $\tau$ is defined in \eqref{eq:defn-tau-case31} and \eqref{eq:defn-tau-case32} for linearly rescaled learning rates and 
 $\tau = 1$ for constant learning rates, and 
 we have also made use of \eqref{eq:eta0_lb_case3}, \eqref{eq:prod-etai-tau-T-23} and \eqref{eq:prod-etai-tau-T-34} in the penultimate inequality in \eqref{eq:prod-etai-tau-T-2345}. 

With \eqref{eqn:death-maiden} in place, we can continue to prove the claim \eqref{eq:sum-LB-1/8-4567}. Recognizing that $\eta_k\big[\prod_{i = k+1}^t \big(1-\eta_i\big)\big]$ is increasing in $k$ (see \eqref{eq:ratio-check_eta_000}),
we can obtain
%
\begin{align}
	\sum_{k = 1}^{\tau-1} \eta_k^2 \Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big]^2 
	&\le \max_{1\leq k < \tau} \left\{ \eta_{k} \Big[\prod_{i = k +1}^t \big(1-\eta_i\big)\Big] \right\} \sum_{k = 1}^{\tau-1} \eta_k \Big[  \prod_{i = k+1}^t \big(1-\eta_i\big) \Big] \notag\\
	&\le \eta_{\tau} \Big[\prod_{i = \tau+1}^t \big(1-\eta_i\big)\Big] \sum_{k = 1}^{\tau-1} \eta_k \Big[  \prod_{i = k+1}^t \big(1-\eta_i\big) \Big] \notag\\
	& \le 7\eta_{\tau} \Big[\prod_{i = \tau+1}^t \big(1-\eta_i\big)\Big] \sum_{k = \tau}^{t} \eta_k\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big],
\end{align}
%
where the last inequality comes from \eqref{eqn:death-maiden}.
With the preceding inequality in place, the claim \eqref{eq:sum-LB-1/8-4567} then follows by observing that 
%
\begin{align*}
	\sum_{k = \tau}^{t} \eta_k^2 \Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big] ^2
	&\ge \min_{\tau\leq k \leq t} \left\{ \eta_{k} \Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big] \right\} \sum_{k = \tau}^{t} \eta_k\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big] \\
	&\ge \eta_{\tau} \Big[\prod_{i = \tau+1}^t \big(1-\eta_i\big)\Big] \sum_{k = \tau}^{t} \eta_k\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big], 
\end{align*}
%
where we make use of the monotonicity of $\eta_k\big[\prod_{i = k+1}^t \big(1-\eta_i\big)\big]$ again.
%
\end{proof}

\begin{proof}[Proof of the claim \eqref{eq:sum-LB-1/8-890}]
%
Note that for $\tau' \defn \max\left\{t - \frac{1}{\eta_{t/2}}, 1\right\}$, one has 
%
\[
\eta_k\Big[\prod_{i = k+1}^t \big(1-\eta_i\big)\Big] \ge \eta_t\big(1-\eta_{t/2}\big)^{t-\tau'} \ge
\eta_t\big(1-\eta_{t/2}\big)^{1/\eta_{t/2}} 
\ge \frac{1}{3}\eta_t,\qquad\text{for all}~\tau'\le k \le t,
\]
%
as long as the following condition holds (recalling the definition of $\widehat{\tau}$ in \eqref{eq:tau-hat-byproduct})
%
\begin{equation}
	\label{eq:eta-t2-t-rank-1/10}
	\eta_{t/2} \le 2\eta_t \le 2\eta_{\widehat{\tau}} \le 1/10.
\end{equation}
%
In addition, 
similar to \eqref{eq:eta-sum-hat-tau}, we can derive
%
\begin{align*}
	\sum_{k = \tau'}^t\eta_k\prod_{i = k+1}^t \big(1-\eta_i\big) &= 1 - \prod_{i = \tau'}^t \big(1-\eta_i\big) 
	\ge 1-\max\left\{\big(1-\eta_t\big)^{1/\eta_{t/2}+1}, \, \prod_{i = 1}^t \big(1-\eta_i\big)\right\} \\
	& \ge 1-\max\left\{e^{-1/2}, \, \prod_{i = 1}^{\widehat{\tau}} \big(1-\eta_i\big)\right\} 
	\ge \frac{1}{8},
\end{align*}
%
where we once again use the condition \eqref{eq:eta-t2-t-rank-1/10}, and the last inequality comes from the derivation in \eqref{eq:prod-etai-tau-T-2345}. 
Putting these two bounds together yields
%
\begin{align*}
\sum_{k=\tau'}^{t}\eta_{k}^{2}\Big[\prod_{i=k+1}^{t}\big(1-\eta_{i}\big)\Big]^{2} & \geq\min_{k:\,\tau'\leq k\leq t}\left\{ \eta_{k}\Big[\prod_{i=k+1}^{t}\big(1-\eta_{i}\big)\Big]\right\} \sum_{k=\tau'}^{t}\eta_{k}\Big[\prod_{i=k+1}^{t}\big(1-\eta_{i}\big)\Big]\\
 & \geq\frac{1}{3}\eta_{t}\cdot\frac{1}{8}\geq\frac{1}{24}\eta_{t}.
\end{align*}
%

To finish up, it remains to justify \eqref{eq:eta-t2-t-rank-1/10}. 
This condition is obvious for constant learning rates. As for rescaled learning rates, one can see that 
%
\[
	\eta_i = \frac{1}{1+(1-\gamma)\ceta i} \ge \frac{19}{20\ceta(1-\gamma)i}  \qquad \text{for all }i \ge \overline{\tau},
\]
%
where $\overline{\tau} \defn \lceil \frac{19}{\ceta(1-\gamma)} \rceil$. This allows one to obtain 
%
\[
\log\Big[\prod_{i = \overline{\tau}}^{T} \big(1-\eta_i(1-\gamma p)\big)\Big] \le -\sum_{i = \overline{\tau}}^{T} \eta_i(1-\gamma p) \le -\sum_{i = \overline{\tau}}^{T}\frac{19}{15\ceta i} \le -\frac{19\log\frac{\,T\,}{\overline{\tau}}}{15\ceta} \le -\frac{19\log\frac{\ceta(1-\gamma)T}{20}}{15\ceta} \le -\frac{1}{5},
\]
%
provided that $T\geq \frac{c_1}{(1-\gamma)^2}$ for some sufficiently large constant $c_1>0$ and $1-\gamma < \ceta < \log T$. 
Taking this together with \eqref{eq:tau-hat-byproduct} implies that $\widehat{\tau} \ge \overline{\tau}$ and hence 
	$\eta_{\widehat{\tau}} \le \eta_{\overline{\tau}} = \frac{1}{1+ (1-\gamma)\ceta \overline{\tau}\,} = 1/20$. 
%
\end{proof}



conclusions.tex
\section{Concluding remarks}

In this paper, we have settled the sample complexity of synchronous Q-learning in $\gamma$-discounted infinite-horizon MDPs, 
which is shown to be 
on the order of $ \widetilde{O}\big( \frac{|\cS|}{(1-\gamma)^3\varepsilon^2} \big) $ when $|\cA|=1$ and $ \widetilde{O}\big( \frac{|\cS||\cA|}{(1-\gamma)^4\varepsilon^2} \big) $ when $|\cA|\geq 2$.
A matching lower bound has been developed when $|\cA|\geq 2$ through studying the dynamics of Q-learning on a hard MDP instance,
which unveils the negative impact of an inevitable over-estimation issue. 
Our theory has been further extended to accommodate asynchronous Q-learning, resulting in tight dependency of the sample complexity on the effective horizon. 
The analysis framework developed herein---which exploits novel error decompositions
and variance control that differ substantially from prior approaches---might suggest a plausible path towards sharpening the sample complexity of, as well as understanding the algorithmic bottlenecks for,  other model-free algorithms (e.g., double Q-learning \citep{hasselt2010double}). 


infinite-synchronous-Q.tex


\section{Upper bounds for Q-learning (Theorem \ref{thm:infinite-horizon-simple})}

\label{sec:Analysis:-infinite-horizon-MDPs}

In this section, we fill in the details for the proof idea outlined in Section \ref{sec:proof-outline-thm:infinite-horizon} for synchronous Q-learning.  
In fact, our proof strategy leads to  a more general version that accounts for
the full $\varepsilon$-range $\varepsilon \in \big(0,\frac{1}{1-\gamma}\big]$, as stated below. 
%
\begin{theorem}\label{thm:infinite-horizon}
Consider any $\gamma\in (0,1)$ and any $\varepsilon\in\big(0,\frac{1}{1-\gamma}\big]$.
Theorem \ref{thm:infinite-horizon-simple} continues to hold if
%
\begin{equation}
	T \ge \frac{c_{3}\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}\min\{\varepsilon^{2},\varepsilon\}}
	\label{eq:sample-size-condition-complete-inf}
\end{equation}
%
for some large enough universal constant $c_{3}>0$. 
\end{theorem}
%
\begin{remark} Clearly, Theorem \ref{thm:infinite-horizon} subsumes Theorem \ref{thm:infinite-horizon-simple} as a special case. \end{remark}


As one can anticipate, the proof of Theorem \ref{thm:infinite-horizon} for Q-learning includes many key ingredients for establishing Theorem~\ref{thm:policy-evaluation} for TD learning. We will elaborate on how to modify the proof argument to establish Theorem~\ref{thm:policy-evaluation} in Section~\ref{sec:TD-learning-analysis}. 




\subsection{Preliminaries}
\label{sec:infinite_prelim}


To begin with, we gather a few elementary facts that shall be used multiple times in the proof. 

\paragraph{Ranges of $\bm{Q}_{t}$ and $\bm{V}_{t}$.} When properly
initialized, the Q-function estimates and the value function estimates
always fall within a suitable range, as asserted by the following
lemma. 

\begin{lemma}
\label{lemma:non-negativity-Qt-Vt}
Suppose that $0\leq\eta_{t}\leq1$
for all $t\geq0$. Assume that $\bm{0}\leq\bm{Q}_{0}\leq\frac{1}{1-\gamma}\bm{1}$.
Then for any $t\geq0$, 
%
\begin{equation}
	\bm{0}\le\bm{Q}_{t}\le\frac{1}{1-\gamma}\bm{1}\qquad\text{and}\qquad\bm{0}\le\bm{V}_{t}\le\frac{1}{1-\gamma}\bm{1}.
	\label{eq:Q-i-LB-UB}
\end{equation}
%
\end{lemma}
%
\begin{proof}
%
We shall prove this by induction. First,
our initialization trivially obeys (\ref{eq:Q-i-LB-UB}) for $t=0$.
Next, suppose that (\ref{eq:Q-i-LB-UB}) is true for the $(t-1)$-th
iteration, namely, 
%
\begin{equation}
	\bm{0}\le\bm{Q}_{t-1}\le\frac{1}{1-\gamma}\bm{1}\qquad\text{and}\qquad\bm{0}\le\bm{V}_{t-1}\le\frac{1}{1-\gamma}\bm{1},
	\label{eq:induction-non-negativity-tminus1}
\end{equation}
%
and we intend to justify the claim for the $t$-th iteration. Recognizing
that $\bm{0}\leq\bm{r}\leq\bm{1}$, $\bm{P}_{t}\geq\bm{0}$ and $\|\bm{P}_{t}\|_{1}=1$,
one can straightforwardly see from the update rule (\ref{eq:iteration-rule-infinite})
and the induction hypothesis \eqref{eq:induction-non-negativity-tminus1}
that
%
\[
\bm{Q}_{t}=(1-\eta_{t})\bm{Q}_{t-1}+\eta_{t}(\bm{r}+\gamma\bm{P}_{t}\bm{V}_{t-1})\geq\bm{0} 
\]
%
and
%
\begin{align*}
\bm{Q}_{t} & =(1-\eta_{t})\bm{Q}_{t-1}+\eta_{t}(\bm{r}+\gamma\bm{P}_{t}\bm{V}_{t-1})\\
 & \leq(1-\eta_{t})\,\|\bm{Q}_{t-1}\|_{\infty}\bm{1}+\eta_{t}\big(\|\bm{r}\|_{\infty}+\gamma\|\bm{P}_{t}\|_{1}\|\bm{V}_{t-1}\|_{\infty}\big)\bm{1}\\
 & \leq(1-\eta_{t})\frac{1}{1-\gamma}\bm{1}+\eta_{t}\Big(1+\frac{\gamma}{1-\gamma}\Big)\bm{1}=\frac{1}{1-\gamma}\bm{1}.
\end{align*}
In addition, from the definition $V_{t}(s):=\max_{a}Q_{t}(s,a)$ for
all $t\geq0$ and all $s\in\mathcal{S}$, it is easily seen that 
\[
\bm{0}\le\bm{V}_{t}\le\frac{1}{1-\gamma}\bm{1},
\]
thus establishing (\ref{eq:Q-i-LB-UB}) for the $t$-th iteration.
Applying the induction argument then concludes the proof. \end{proof}

As a result of Lemma \ref{lemma:non-negativity-Qt-Vt}
and the fact $\bm{0}\le\bm{Q}^{\star}\le\frac{1}{1-\gamma}\bm{1}$,
we have
%
\begin{equation}
\|\bm{Q}_{t}-\bm{Q}^{\star}\|_{\infty}\le\frac{1}{1-\gamma}\qquad\text{for all }t\geq0, \label{eq:Delta-t-range-infinite}
\end{equation}
%
which also confirms that $0\leq \varepsilon\leq\frac{1}{1-\gamma}$ is the full $\varepsilon$-range we need to consider. 
Further, we make note of a direct
consequence of the claimed iteration number (\ref{eq:sample-size-condition-complete-inf})
when $\varepsilon\leq\frac{1}{1-\gamma}$: 
%
\begin{equation}
	T=\frac{c_{3}\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{4}\min\{\varepsilon,\varepsilon^{2}\}}
	\geq\frac{c_{3}\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{3}},\label{eq:T-bound-infinite}
\end{equation}
%
which will be useful for subsequent analysis. 

\paragraph{Several facts regarding the learning rates.} Next, we
gather a couple of useful bounds regarding the learning rates $\{\eta_{t}\}$.
To begin with, we find it helpful to introduce the following related
quantities introduced previously in \eqref{def:eta-i-t-simple}:
%
\begin{equation}
\eta_{i}^{(t)}\coloneqq\begin{cases}
\prod_{j=1}^{t}(1-\eta_{j}), & \text{if }i=0,\\
\eta_{i}\prod_{j=i+1}^{t}(1-\eta_{j}), & \text{if }0<i<t,\\
\eta_{t}, & \text{if }i=t. 
\end{cases}\label{def:eta-i-t}
\end{equation}
%
%whose importance will be made clear momentarily. 
We now take a moment to bound $\eta_{i}^{(t)}$. From our assumption \eqref{eq:thm:infinite-horizaon-eta}
and the condition \eqref{eq:T-bound-infinite}, we know that the learning
rate obeys 
\begin{equation}
\frac{1}{2c_{1}(1-\gamma)T/\log^{3}T}\leq\frac{1}{1+c_{1}(1-\gamma)T/\log^{3}T}\le\eta_{t}\le\frac{1}{1+c_{2}(1-\gamma)t/\log^{3}T}\leq\frac{1}{c_{2}(1-\gamma)t/\log^{3}T}\label{eq:eta-t-UB-LB-inf}
\end{equation}
for some constants $c_{1},c_{2}>0$. Recalling that
%
\begin{equation}
\label{eq:defn-beta-c4}
\beta\coloneqq\frac{c_{4}(1-\gamma)}{\log T}
\end{equation}
%
for some universal constant $c_{4}>0$ and considering any $t$ obeying
%
\begin{equation}
t\ge\frac{T}{c_{2}\log T},\label{eq:t-LB-c2-inf}
\end{equation}
%
we shall bound $\eta_{i}^{(t)}$ by looking at two cases separately. 
%
\begin{itemize}
%
\item For any $0\leq i\le(1-\beta)t$, we can use \eqref{eq:eta-t-UB-LB-inf}
% and the fact $\log T\geq2\log\frac{1}{1-\gamma}$ (see Condition \eqref{eq:T-bound-infinite})
to show that
%
\begin{subequations}\label{eq:eta-i-t-UB12}
%
\begin{align}
\eta_{i}^{(t)} & \le\Big(1-\frac{1}{2c_{1}(1-\gamma)T/\log^{3}T}\Big)^{\beta t}\leq\Big(1-\frac{1}{2c_{1}(1-\gamma)T/\log^{3}T}\Big)^{\frac{c_{4}(1-\gamma)T}{c_{2}\log^{2}T}} \notag\\
 & =\Bigg(\Big(1-\frac{1}{2c_{1}(1-\gamma)T/\log^{3}T}\Big)^{\frac{2c_{1}(1-\gamma)T}{\log^{3}T}}\Bigg)^{\frac{c_{4}\log T}{2c_{1}c_{2}}}<\frac{1}{2T^{2}},
	\label{eq:eta-i-t-UB1}
\end{align}
%
where the last inequality holds as long as $c_{1}c_{2}\le c_{4}/8$. 

\item When it comes to the case with $i>(1-\beta)t\geq t/2$, one can upper bound
%
\begin{equation}
\eta_{i}^{(t)}\le\eta_{i}\leq\frac{1}{c_{2}(1-\gamma)i/\log^{3}T}<\frac{2}{c_{2}(1-\gamma)t/\log^{3}T}\leq\frac{2\log^{4}T}{(1-\gamma)T},\label{eq:eta-i-t-UB2}
\end{equation}
%
\end{subequations}
%
where we have used the constraint \eqref{eq:t-LB-c2-inf}. 
\end{itemize}
%
Moreover, the sum of $\eta_{i}^{(t)}$ over $i$ obeys
%
\begin{align}
\sum_{i=0}^{t}\eta_{i}^{(t)} & =\prod_{j=1}^{t}(1-\eta_{j})+\eta_{1}\prod_{j=2}^{t}(1-\eta_{j})+\eta_{2}\prod_{j=3}^{t}(1-\eta_{j})+\cdots+\eta_{t-1}(1-\eta_{t})+\eta_{t}\nonumber \\
 & =\prod_{j=2}^{t}(1-\eta_{j})+\eta_{2}\prod_{j=3}^{t}(1-\eta_{j})+\cdots+\eta_{t-1}(1-\eta_{t})+\eta_{t}=\cdots\nonumber \\
 & =(1-\eta_{t})+\eta_{t}=1.
	\label{eq:sum-eta-i-t-infinite}
\end{align}
%
Repeating the same argument further allows us to derive
%
\begin{align}
\sum_{i=\tau}^{t}\eta_{i}^{(t)} & = 1 - \prod_{j=\tau}^{t}(1-\eta_{j})
	\label{eq:sum-eta-i-t-infinite-tau}
\end{align}
%
for any $\tau \leq t$.


% \begin{comment}
% To see this, we know that
% \[
% Q_{t-1}(s,\pi^{\star}(s))-Q^{\star}(s,\pi^{\star}(s))\le V_{t-1}(s)-V^{\star}(s)\le Q_{t-1}(s,\pi_{t-1}(s))-Q^{\star}(s,\pi_{t-1}(s))
% \]
% holds for all $s$. Since $\bm{P}$ is a probability transition kernel,
% (\ref{eq:V2Q-infinite}) immediately follows from the simple fact
% that $\bm{P}\bm{a}\leq\bm{P}\bm{b}$ for any vectors $\bm{a}\leq\bm{b}$.
% \end{comment}


% \subsection{Proof details of Theorem \ref{thm:infinite-horizon}}


\subsection{Proof of Lemma~\ref{lem:inf-upper-bound} }
\label{sec:proof_inf_upper_bound}
% We shall begin by upper bounding $\bm{\Delta}_{t}$ for $t\ge\frac{T}{c_{2}\log T}$.


%Recalling the definition \eqref{eq:defn-beta-c4} of $\beta$, and one further decomposes the upper bound
%$$\bm{\Delta}_{t}  \le\eta_{0}^{(t)}\bm{\Delta}_{0}+\sum_{i=1}^{t}\eta_{i}^{(t)}\gamma\big\{\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}+(\bm{P}_{i}-\bm{P})\bm{V}_{i-1}\big\} $$
%one
%as follows
%\begin{align}
%\bm{\Delta}_{t} & \leq\underbrace{\eta_{0}^{(t)}\bm{\Delta}_{0}+\sum_{i=1}^{(1-\beta)t}\eta_{i}^{(t)}\gamma\big(\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}+(\bm{P}_{i}-\bm{P})\bm{V}_{i-1}\big)}_{=:\,\bm{\zeta}_{t}}\nonumber \\
% & \quad\quad+\underbrace{\sum_{i=(1-\beta)t+1}^{t}\eta_{i}^{(t)}\gamma(\bm{P}_{i}-\bm{P})\bm{V}_{i-1}}_{=:\,\bm{\xi}_{t}}+\sum_{i=(1-\beta)t+1}^{t}\eta_{i}^{(t)}\gamma\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}.\label{eq:Delta-UB2}
%\end{align}


We shall exploit the relation \eqref{eq:Delta-UB2} to prove this lemma. 
One of the key ingredients of our analysis lies in controlling the terms $\bm{\zeta}_{t}$
and $\bm{\xi}_{t}$ introduced in  \eqref{eq:Delta-UB2}, which in turn enables us to apply (\ref{eq:Delta-UB2})
recursively to control $\bm{\Delta}_{t}$. 

\paragraph{Step 1: bounding $\bm{\zeta}_{t}$.} 

We start by developing an upper bound on $\bm{\zeta}_{t}$ (cf.~\eqref{eq:Delta-UB2}) for
any $t$ obeying $\frac{T}{c_{2}\log T}\leq t\leq T$.
Invoking the preceding upper bounds \eqref{eq:eta-i-t-UB12} on $\eta_{i}^{(t)}$
implies that
%
\begin{align*}
\|\bm{\zeta}_{t}\|_{\infty} & \leq\eta_{0}^{(t)}\|\bm{\Delta}_{0}\|_{\infty}+t\max_{i\leq(1-\beta)t}\eta_{i}^{(t)}\max_{1\leq i\leq(1-\beta)t}\big(\|\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}\|_{\infty}+\|\bm{P}_{i}\bm{V}_{i-1}\|_{\infty}+\|\bm{P}\bm{V}_{i-1}\|_{\infty}\big)\\
 & \leq\eta_{0}^{(t)}\|\bm{\Delta}_{0}\|_{\infty}+t\max_{i\leq(1-\beta)t}\eta_{i}^{(t)}\max_{1\leq i\leq(1-\beta)t}\Big\{\|\bm{P}^{\pi_{i-1}}\|_{1}\|\bm{\Delta}_{i-1}\|_{\infty}+\big(\|\bm{P}_{i}\|_{1}+\|\bm{P}\|_{1}\big)\|\bm{V}_{i-1}\|_{\infty}\Big\}\\
 & \overset{(\mathrm{i})}{=}\eta_{0}^{(t)}\|\bm{\Delta}_{0}\|_{\infty}+t\max_{i\leq(1-\beta)t}\eta_{i}^{(t)}\max_{1\leq i\leq(1-\beta)t}\big(\|\bm{\Delta}_{i-1}\|_{\infty}+2\,\|\bm{V}_{i-1}\|_{\infty}\big)\\
 & \overset{(\mathrm{ii})}{\leq}\frac{1}{2T^{2}}\cdot\frac{1}{1-\gamma}+\frac{1}{2T^{2}}\cdot t\cdot\frac{3}{1-\gamma}\\
 & \le\frac{2}{(1-\gamma)T}.
\end{align*}
%
Here, (i) holds since $\|\bm{P}^{\pi_{i-1}}\|_{1}=\|\bm{P}_{i}\|_{1}=\|\bm{P}\|_{1}=1$
(as they are all probability transition matrices), whereas (ii) arises
from the previous bound (\ref{eq:eta-i-t-UB1}).



\paragraph{Step 2: bounding $\bm{\xi}_{t}$.} 

Moving on to the term $\bm{\xi}_{t}$, let us express it as
%
\[
\bm{\xi}_{t}=\sum_{i=(1-\beta)t+1}^{t}\bm{z}_{i}\qquad\text{with }\bm{z}_{i}\coloneqq\eta_{i}^{(t)}\gamma(\bm{P}_{i}-\bm{P})\bm{V}_{i-1},
\]
%
where the $\bm{z}_{i}$'s satisfy
%
\[
\mathbb{E}\left[\bm{z}_{i}\mymid\bm{V}_{i-1},\cdots,\bm{V}_{0}\right]=\bm{0}.
\]
%
This motivates us to invoke Freedman's inequality (see Theorem \ref{thm:Freedman})
to control $\bm{\xi}_{t}$ for any $t$ obeying $\frac{T}{c_{2}\log T}\leq t\leq T$.
Towards this, we need to calculate several quantities.
\begin{itemize}
\item First, it is seen that
\begin{align*}
B & :=\max_{(1-\beta)t<i\leq t}\|\bm{z}_{i}\|_{\infty}\leq\max_{(1-\beta)t<i\leq t}\|\eta_{i}^{(t)}(\bm{P}_{i}-\bm{P})\bm{V}_{i-1}\|_{\infty}\\
 & \leq\max_{(1-\beta)t<i\leq t}\eta_{i}^{(t)}\big(\|\bm{P}_{i}\|_{1}+\|\bm{P}\|_{1}\big)\|\bm{V}_{i-1}\|_{\infty}\leq\frac{4\log^{4}T}{(1-\gamma)^{2}T},
\end{align*}
where the last inequality is due to (\ref{eq:eta-i-t-UB2}), Lemma~\ref{lemma:non-negativity-Qt-Vt},
and the fact $\|\bm{P}_{i}\|_{1}=\|\bm{P}\|_{1}=1$. 


\item Next, we turn to certain variance terms.  For any vector $\bm{a}=[a_j]$, let us use $\mathsf{Var}\big(\bm{a}\mymid\bm{V}_{i-1},\cdots,\bm{V}_{0}\big)$
to denote a vector whose $j$-th entry is given by $\mathsf{Var}\big(a_{j}\mymid\bm{V}_{i-1},\cdots,\bm{V}_{0}\big)$.
With this notation in place, and recalling the notation $\mathsf{Var}_{\bm{P}}(\bm{z})$ in (\ref{eq:defn-Var-P-V}), we obtain 
%
\begin{align}
\bm{W}_{t} & :=\sum_{i=(1-\beta)t+1}^{t}\mathsf{Var}\big(\bm{z}_{i}\mymid\bm{V}_{i-1},\cdots,\bm{V}_{0}\big) 
	=\gamma^{2}\sum_{i=(1-\beta)t+1}^{t}\big(\eta_{i}^{(t)}\big)^{2}\mathsf{Var}\Big( (\bm{P}_{i}-\bm{P})\bm{V}_{i-1} \mymid\bm{V}_{i-1} \Big) \nonumber \\
 & =\gamma^{2}\sum_{i=(1-\beta)t+1}^{t}\big(\eta_{i}^{(t)}\big)^{2}\mathsf{Var}_{\bm{P}}\big(\bm{V}_{i-1}\big)\nonumber \\
 & \le\Big(\max_{(1-\beta)t\leq i\leq t}\eta_{i}^{(t)}\Big)\Big(\sum_{i=(1-\beta)t+1}^{t}\eta_{i}^{(t)}\Big)\max_{(1-\beta)t\leq i<t}\mathsf{Var}_{\bm{P}}\big(\bm{V}_{i}\big)\nonumber \\
 & \leq\frac{2\log^{4}T}{(1-\gamma)T}\max_{(1-\beta)t\leq i<t}\mathsf{Var}_{\bm{P}}(\bm{V}_{i}),\label{eq:Wt-bound-Var-infinite}
\end{align}
%
where the last inequality relies on the previous bounds \eqref{eq:eta-i-t-UB2} and \eqref{eq:sum-eta-i-t-infinite}.
 
\item In the meantime, Theorem~\ref{lemma:non-negativity-Qt-Vt} leads us to the following
trivial upper bound:
\[
\big|\bm{W}_{t}\big|\leq\frac{2\log^{4}T}{(1-\gamma)T}\cdot\frac{1}{(1-\gamma)^2}\bm{1}
=\frac{2\log^{4}T}{(1-\gamma)^{3}T}\bm{1}\eqqcolon\sigma^{2}\bm{1}.
\]
By setting $K=\left\lceil 2\log_2\frac{1}{1-\gamma}\right\rceil $, one has
%
\begin{equation}
\frac{\sigma^{2}}{2^{K}}\leq\frac{2\log^{4}T}{(1-\gamma)T}.\label{eq:sigma-2K-UB-inf}
\end{equation}
%
\end{itemize}
With the above bounds in place, applying the Freedman inequality in
Theorem \ref{thm:Freedman} and invoking the union bound over all the $|\cS||\cA|$ entries
of $\bm{\xi}_{t}$ demonstrate that 
%
\begin{align*}
|\bm{\xi}_{t}| & \le
\sqrt{8\Big(\bm{W}_{t}+\frac{\sigma^{2}}{2^{K}}\bm{1}\Big)\log\frac{8|\mathcal{S}||\mathcal{A}|T\log\frac{1}{1-\gamma}}{\delta}}+\Big(\frac{4}{3}B\log\frac{8|\mathcal{S}||\mathcal{A}|T\log\frac{1}{1-\gamma}}{\delta}\Big)\cdot\bm{1}\\
 & \le\sqrt{16\Big(\bm{W}_{t}+\frac{2\log^{4}T}{(1-\gamma)T}\bm{1}\Big)\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}}+\Big(3B\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\Big)\cdot\bm{1}\\
 & \le\sqrt{\frac{32\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)T}\Big(\max_{(1-\beta)t\leq i<t}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})+\bm{1}\Big)}+\frac{12\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{2}T}\bm{1}
\end{align*}
%
with probability at least $1-\delta/T.$ Here, the second line holds
due to \eqref{eq:sigma-2K-UB-inf} and the fact $\log\frac{8|\mathcal{S}||\mathcal{A}|T\log\frac{1}{1-\gamma}}{\delta}\le2\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}$
(cf.~\eqref{eq:T-bound-infinite}), whereas the last inequality makes
use of the relation \eqref{eq:Wt-bound-Var-infinite}. 



\paragraph{Step 3: using the bounds on $\bm{\zeta}_{t}$ and $\bm{\xi}_{t}$
to control $\bm{\Delta}_{t}$.} Let us define
%
\begin{equation}
\bm{\varphi}_{t}\coloneqq 64\frac{\log^{4}T\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}}{(1-\gamma)T}\bigg(\max_{\frac{t}{2}\le i\leq t}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})+\bm{1}\bigg)\label{def:phi}
\end{equation}
%
%for some sufficiently large constant $c_{5}>0$. 
In view of the upper bounds derived in Steps 1 and 2, and $\beta$ defined in \eqref{eq:defn-beta-c4}, 
we have --- with probability exceeding
$1-\delta$ --- that
%
\begin{equation}
|\bm{\zeta}_{k}|+|\bm{\xi}_{k}|\leq\sqrt{\bm{\varphi}_{t}}\qquad\text{for all }~2t/3\le k\le t, \label{eq:zeta-xi-UB}
\end{equation}
%
provided that $T\geq \frac{c_9 (\log^4 T)\big( \log \frac{|\cS||\cA|T}{\delta}\big) }{ (1-\gamma)^3} $ for some sufficiently large constant $c_9>0$.
Substituting (\ref{eq:zeta-xi-UB}) into (\ref{eq:Delta-UB2}), we
can upper bound $\bm{\Delta}_{t}$ as follows
\begin{align}
\bm{\Delta}_{k} & \le\sqrt{\bm{\varphi}_{t}}+\sum_{i=(1-\beta)k+1}^{k}\eta_{i}^{(k)}\gamma\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}=\sqrt{\bm{\varphi}_{t}}+\sum_{i=(1-\beta)k}^{k-1}\eta_{i+1}^{(k)}\gamma\bm{P}^{\pi_{i}}\bm{\Delta}_{i}\qquad\text{for all }~2t/3\le k\le t.\label{eq:Deltak-UB-phit-infinite}
\end{align}


Further, we find it convenient to define $\big\{\alpha_{i}^{(t)}\big\}$
as follows
%
\begin{align}
\alpha_{i}^{(t)}:=\frac{\eta_{i+1}^{(t)}}{\sum_{j=(1-\beta)t}^{t-1}\eta_{j+1}^{(t)}}.
	\label{eq:defn-alpha-it-sync}
\end{align}
%
Clearly, this sequence satisfies
%
\begin{equation}
\alpha_{i}^{(t)}\ge\eta_{i+1}^{(t)}\qquad\text{and}\qquad\sum_{i=(1-\beta)t}^{t-1}\alpha_{i}^{(t)}=1\label{eq:relation-alpha-eta-inf}
\end{equation}
%
for any $t$, where the first inequality results from \eqref{eq:sum-eta-i-t-infinite}.
With these in place, we can write \eqref{eq:Deltak-UB-phit-infinite} as
%
\begin{equation}
\bm{\Delta}_{k}\le\sqrt{\bm{\varphi}_{t}}+\sum_{i_{1}=(1-\beta)k}^{k-1}\eta_{i_{1}+1}^{(k)}\gamma\bm{P}^{\pi_{i_{1}}}\bm{\Delta}_{i_{1}}=\sum_{i_{1}=(1-\beta)k}^{k-1}\Big(\alpha_{i_{1}}^{(k)}\sqrt{\bm{\varphi}_{t}}+\eta_{i_{1}+1}^{(k)}\gamma\bm{P}^{\pi_{i_{1}}}\bm{\Delta}_{i_{1}}\Big)\qquad\text{for all }2t/3\le k\le t.\label{eq:Delta-k-recursive-inf}
\end{equation}
%
Given that $(1-\beta)t\geq2t/3$ (see \eqref{eq:defn-beta-c4}), we
can invoke this relation recursively to yield
%
\begin{align}
\bm{\Delta}_{t} & \le\sum_{i_{1}=(1-\beta)t}^{t-1}\Big(\alpha_{i_{1}}^{(t)}\sqrt{\bm{\varphi}_{t}}+\eta_{i_{1}+1}^{(t)}\gamma\bm{P}^{\pi_{i_{1}}}\bm{\Delta}_{i_{1}}\Big)\nonumber \\
 & \leq\sum_{i_{1}=(1-\beta)t}^{t-1}\Bigg[\alpha_{i_{1}}^{(t)}\sqrt{\bm{\varphi}_{t}}+\eta_{i_{1}+1}^{(t)}\gamma\bm{P}^{\pi_{i_{1}}}\sum_{i_{2}=(1-\beta)i_{1}}^{i_{1}-1}\Big(\alpha_{i_{2}}^{(i_{1})}\sqrt{\bm{\varphi}_{t}}+\eta_{i_{2}+1}^{(i_{1})}\gamma\bm{P}^{\pi_{i_{2}}}\bm{\Delta}_{i_{2}}\Big)\Bigg]\nonumber \\
 & \leq\sum_{i_{1}=(1-\beta)t}^{t-1}\alpha_{i_{1}}^{(t)}\sqrt{\bm{\varphi}_{t}}+\sum_{i_{1}=(1-\beta)t}^{t-1}\sum_{i_{2}=(1-\beta)i_{1}}^{i_{1}-1}\alpha_{i_{1}}^{(t)}\alpha_{i_{2}}^{(i_{1})}\big(\gamma\bm{P}^{\pi_{i_{1}}}\big)\sqrt{\bm{\varphi}_{t}} \nonumber \\
 & \qquad\qquad +\sum_{i_{1}=(1-\beta)t}^{t-1}\sum_{i_{2}=(1-\beta)i_{1}}^{i_{1}-1}\eta_{i_{1}+1}^{(t)}\eta_{i_{2}+1}^{(i_{1})}\prod_{k=1}^{2}\big(\gamma\bm{P}^{\pi_{i_{k}}}\big)\bm{\Delta}_{i_{2}}\nonumber \\
 & =\sum_{i_{1}=(1-\beta)t}^{t-1}\sum_{i_{2}=(1-\beta)i_{1}}^{i_{1}-1}\alpha_{i_{1}}^{(t)}\alpha_{i_{2}}^{(i_{1})}\left\{ \bm{I}+\gamma\bm{P}^{\pi_{i_{1}}}\right\} \sqrt{\bm{\varphi}_{t}}+\sum_{i_{1}=(1-\beta)t}^{t-1}\sum_{i_{2}=(1-\beta)i_{1}}^{i_{1}-1}\eta_{i_{1}+1}^{(t)}\eta_{i_{2}+1}^{(i_{1})}\prod_{k=1}^{2}\big(\gamma\bm{P}^{\pi_{i_{k}}}\big)\bm{\Delta}_{i_{2}},\label{eq:Delta-recursion-inf}
\end{align}
%
where the second inequality relies on \eqref{eq:Delta-k-recursive-inf}, the third line uses the inequality $\eta_{i_{1}+1}^{(t)}\leq\alpha_{i_{1}}^{(t)}$ in \eqref{eq:relation-alpha-eta-inf}, and the fourth line is valid
since $\sum_{i_{2}=(1-\beta)i_{1}}^{i_{1}-1}\alpha_{i_{2}}^{(i_{1})}=1$
(see \eqref{eq:relation-alpha-eta-inf}). 

We intend to continue invoking \eqref{eq:Delta-k-recursive-inf}
recursively  --- similar to how we derive \eqref{eq:Delta-recursion-inf} --- in
order to control $\bm{\Delta}_{t}$. To do so, we are in need of some
preparation. First, let us define
\begin{equation}
H\coloneqq\frac{\log T}{1-\gamma}\qquad\text{and}\qquad\alpha_{\{i_{k}\}_{k=1}^{H}}:=\alpha_{i_{1}}^{(t)}\alpha_{i_{2}}^{(i_{1})}\ldots\alpha_{i_{H}}^{(i_{H-1})}\ge0\label{eq:defn-H-infinite}
\end{equation}
for any $t>i_{1}>i_{2}>\cdots>i_{H}$, which clearly satisfies (see
\eqref{eq:relation-alpha-eta-inf})

\begin{equation}
\alpha_{\{i_{k}\}_{k=1}^{H}}\geq\eta_{i_{1}+1}^{(t)}\eta_{i_{2}+1}^{(i_{1})}\ldots\eta_{i_{H}+1}^{(i_{H-1})}.\label{eq:alpha-set-relation-inf-1}
\end{equation}
In addition, defining the index set 
\begin{equation}
\mathcal{I}_{t}\coloneqq\Big\{\left(i_{1},\cdots,i_{H}\right)\mymid(1-\beta)t\leq i_{1}\leq t-1,\ \forall1\leq j<H:(1-\beta)i_{j}\leq i_{j+1}\leq i_{j}-1\Big\},\label{eq:defn-It-inf}
\end{equation}
we have
\begin{equation}
\sum_{\left(i_{1},\cdots,i_{H}\right)\in\mathcal{I}_{t}}\alpha_{\{i_{k}\}_{k=1}^{H}}=1.\label{eq:alpha-set-relation-inf}
\end{equation}
Additionally, recalling that $\beta=c_{4}(1-\gamma)/\log T$, we see
that this choice of $H$ satisfies 
%
\[
(1-\beta)^{H}=\left(1-\frac{c_{4}(1-\gamma)}{\log T}\right)^{\frac{\log T}{1-\gamma}}\geq\frac{2}{3}
\]
%
 for $c_{4}$ small enough, thus implying that
 %
\[
i_{1}>i_{2}>\cdots>i_{H}\geq(1-\beta)^{H}t\geq2t/3\qquad\text{for all }(i_{1},\cdots,i_{H})\in\mathcal{I}_{t}.
\]
%
This is an important property that allows one to invoke the relation
\eqref{eq:Delta-k-recursive-inf}. With these in place, applying the
preceding relation \eqref{eq:Delta-k-recursive-inf} recursively ---
in a way similar to \eqref{eq:Delta-recursion-inf} --- further leads to
%
\begin{align}
\bm{\Delta}_{t} & \le\sum_{\left(i_{1},\cdots,i_{H}\right)\in\mathcal{I}_{t}}\alpha_{\{i_{k}\}_{k=1}^{H}}\Bigg\{\bigg(\bm{I}+\sum_{h=1}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\bigg)\sqrt{\bm{\varphi}_{t}}+\gamma^{H}\prod_{k=1}^{H}\bm{P}^{\pi_{i_{k}}}\big|\bm{\Delta}_{i_{H}}\big|\Bigg\}\nonumber \\
 & \leq\max_{\left(i_{1},\cdots,i_{H}\right)\in\mathcal{I}_{t}}\Bigg\{\underbrace{\bigg(\bm{I}+\sum_{h=1}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\bigg)\sqrt{\bm{\varphi}_{t}}}_{=:\,\bm{\beta}_{1}}+\underbrace{\gamma^{H}\prod_{k=1}^{H}\bm{P}^{\pi_{i_{k}}}\big|\bm{\Delta}_{i_{H}}\big|}_{=:\,\bm{\beta}_{2}}\Bigg\}\label{eq:Delta-UB-temp-inf}
\end{align}
%
for all $t\ge\frac{T}{c_{2}\log T}$, where we recall the definition of the entrywise $\max$ operator in Section~\ref{subsec:matrix-notation}. 
Here, the last inequality relies on the fact that $\sum_{\left(i_{1},\cdots,i_{H}\right)\in\mathcal{I}_{t}}\alpha_{\{i_{k}\}_{k=1}^{H}}=1$ (see \eqref{eq:alpha-set-relation-inf}). 
It remains to control $\bm{\beta}_{1}$ and $\bm{\beta}_{2}$, which we shall accomplish separately in the next two steps. 



\paragraph{Step 4: bounding $\bm{\beta}_{2}$.}

%It remains to control $\bm{\beta}_{1}$ and $\bm{\beta}_{2}$, which we shall accomplish separately as follows. 


%\begin{itemize}
%\item 

The term $\bm{\beta}_{2}$ defined in \eqref{eq:Delta-UB-temp-inf} is relatively easier to control. Observing that $\prod_{k=1}^{H}\bm{P}^{\pi_{i_{k}}}$
is still a probability transition matrix, we can derive
%
\begin{align*}
|\bm{\beta}_{2}| & =\gamma^{H}\prod_{1\leq k\leq h}\bm{P}^{\pi_{i_{k}}}\big|\bm{\Delta}_{i_{H}}\big|\le\gamma^{H}\bigg\|\prod_{1\leq k\leq h}\bm{P}^{\pi_{i_{k}}}\bigg\|_{1}\|\bm{\Delta}_{i_{H}}\|_{\infty}=\gamma^{H}\|\bm{\Delta}_{i_{H}}\|_{\infty}\\
 & \overset{(\mathrm{i})}{\leq}\frac{1}{1-\gamma}\gamma^{H}\overset{(\mathrm{ii})}{\leq}\frac{1}{(1-\gamma)T},
\end{align*}
%
where (i) results from the crude bound (\ref{eq:Delta-t-range-infinite}).
To justify the inequality (ii), we recall the definition \eqref{eq:defn-H-infinite}
of $H$ to see that
%
\[
\gamma^{H}\overset{(\mathrm{ii})}{=}\big(1-(1-\gamma)\big)^{\frac{1}{1-\gamma}\log T}\leq e^{-\log T}=\frac{1}{T},
\]
where the inequality comes from the elementary fact that $\gamma^{\frac{1}{1-\gamma}}\leq e^{-1}$
for any $0<\gamma<1$.


\paragraph{Step 5: bounding $\bm{\beta}_{1}$.}

When it comes to the term $\bm{\beta}_{1}$ defined in \eqref{eq:Delta-UB-temp-inf}, we can upper bound the entrywise
square of $\bm{\beta}_{1}$ --- denoted by $|\bm{\beta}_{1}|^{2}$
--- as follows
%
\begin{align}
	|\bm{\beta}_{1}|^{2} & =\bigg| \Big(  \sum_{h=0}^{H-1}\gamma^{h}\prod_{1\leq k\leq h}\bm{P}^{\pi_{i_{k}}} \Big)\sqrt{\bm{\varphi}_{t}}\bigg|^{2}\overset{(\mathrm{i})}{\leq}\bigg|\sum_{h=0}^{H-1}\gamma^{h/2}\cdot\gamma^{h/2}\sqrt{\prod_{1\leq k\leq h}\bm{P}^{\pi_{i_{k}}}\bm{\varphi}_{t}}\,\bigg|^{2}\nonumber \\
 & \overset{(\mathrm{ii})}{\leq}\sum_{h=0}^{H-1}\gamma^{h}\cdot\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\bm{\varphi}_{t}\nonumber \\
 & \overset{(\mathrm{iii})}{\leq}\frac{1}{1-\gamma}\sum_{h=0}^{H-1} \gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\frac{64\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)T}\bigg(\max_{\frac{t}{2}\le i<t}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})+\bm{1}\bigg)\nonumber \\
 & \overset{(\mathrm{iv})}{\leq}\frac{64\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{2}T}\sum_{h=0}^{H-1} \gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\max_{\frac{t}{2}\le i<t}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})+\frac{64\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{3}T}\bm{1}\nonumber.
\end{align}
%
Here, (i) follows from Jensen's inequality and the fact that $\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}$
is a probability transition matrix; (ii) holds due to the Cauchy-Schwarz
inequality; (iii) utilizes the definition of $\bm{\varphi}_{t}$ in
(\ref{def:phi}); (iv) follows since $\prod_{1\leq k\leq h}\bm{P}^{\pi_{i_{k}}}\bm{1}=\bm{1}$
and $\sum_{0\leq h<H}\gamma^{h}\leq\frac{1}{1-\gamma}$.
To further control the right-hand side of the above inequality, we resort to the following lemma.  
%
\begin{lemma}\label{lemma:inf-alpha-claim}
Suppose that $t\ge\frac{T}{c_{2}\log T}$. For any $\left(i_{1},\cdots,i_{H}\right)\in\mathcal{I}_{t}$, 
the following holds:
%
\begin{equation}
\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\max_{\frac{t}{2}\le i<t}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})
	\leq \frac{4}{\gamma^2(1-\gamma)^{2}}\Big(1+2\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\Big)\bm{1}.\label{eq:inf-alpha-claim}
\end{equation}
%
\end{lemma}
\begin{proof} This lemma, which is inspired by but significantly more complicated than \citet[Lemma 8]{azar2013minimax},
plays a key role in shaving one $\frac{1}{1-\gamma}$ factor. 
See Section~\ref{subsec:Proof-of-eq:inf-alpha-claim} for the proof.\end{proof}
%
Therefore, the above result directly implies that 
%
\begin{align}
|\bm{\beta}_{1}|^{2} 
& \leq 
\frac{320\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2 (1-\gamma)^{4}T}\Big(1+2\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\Big)\bm{1}.\label{eq:alpha-1-UB}
\end{align}
%


\paragraph{Step 6: putting all this together.}
Substituting the preceding bounds for $\bm{\beta}_{1}$ and $\bm{\beta}_{2}$
into \eqref{eq:Delta-UB-temp-inf}, we can demonstrate that: with
probability at least $1-\delta$, 
\begin{align}
\bm{\Delta}_{t} & \leq \frac{1}{(1-\gamma)T}\bm{1}+\sqrt{\frac{320\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2 (1-\gamma)^{4}T}\Big(1+2\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\Big)}\ \bm{1}\nonumber \\
 & \leq 30 \sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2 (1-\gamma)^{4}T}\Big(1+\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\Big)}\ \bm{1}\label{eq:Delta-UB-final}
\end{align}
holds simultaneously for all $t\ge\frac{T}{c_{2}\log T}$,
where the second line is valid since $\frac{1}{(1-\gamma)T}\leq \sqrt{\frac{(\log^{4}T)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{4}T}}$
under our sample size condition \eqref{eq:T-bound-infinite}. 




\subsection{Proof of Lemma~\ref{lem:inf-lower-bound} }
\label{sec:proof_inf_lower_bound}

Next, we move forward to develop an lower bound on $\bm{\Delta}_{t}$,
which can be accomplished in an analogous manner as for the above
upper bound. Applying a similar argument for (\ref{eq:Delta-UB-temp-inf})
(except that we need to replace $\pi_{i}$ with $\pi^{\star}$), one
can deduce that
\begin{align}
\bm{\Delta}_{t} & \ge-\max_{\left(i_{1},\cdots,i_{H}\right)\in\mathcal{I}_{t}}\left\{\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi^{\star}}\sqrt{\bm{\varphi}_{t}}+\gamma^{H}\prod_{k=1}^{H}\bm{P}^{\pi^{\star}}\big|\bm{\Delta}_{i_{H}}\big|\right\}\label{eq:Delta-t-lower-bound-inf}
\end{align}
for any $t\ge\frac{c_{2}T}{\log\frac{1}{1-\gamma}}$. It is straightforward
to bound the second term on the right-hand side of \eqref{eq:Delta-t-lower-bound-inf}
as
\[
\gamma^{H}\prod_{1\leq k\leq H}\bm{P}^{\pi^{\star}}\big|\bm{\Delta}_{i_{H}}\big|\le\gamma^{H}\bigg\|\prod_{1\leq k\leq H}\bm{P}^{\pi^{\star}}\bigg\|_{1}\|\bm{\Delta}_{i_{H}}\|_{\infty}\bm{1}\le\frac{1}{(1-\gamma)T}\bm{1},
\]
where the second inequality makes use of \eqref{eq:Delta-t-range-infinite}
as well as the fact that $\prod_{k}\bm{P}^{\pi^{\star}}$ is a probability
transition matrix (so that $\|\prod_{k}\bm{P}^{\pi^{\star}}\big\|_{1}=1$).
As for the first term on the right-hand side of \eqref{eq:Delta-t-lower-bound-inf},
we can invoke a similar argument for (\ref{eq:alpha-1-UB}) to obtain
\begin{align*}
\bigg|\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi^{\star}}\sqrt{\bm{\varphi}_{t}}\bigg|^{2} & 
\leq 320\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\bigg(1+2\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\bigg)\bm{1}.
\end{align*}
%
Taking these two bounds together, we see that with probability at
least $1-\delta$, 
%
\begin{equation}
\bm{\Delta}_{t}\geq -30 \sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2 (1-\gamma)^{4}T}\bigg(1+\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\bigg)}\ \bm{1}\label{eq:Delta-LB-final}
\end{equation}
holds simultaneously for all $t\ge\frac{T}{c_{2}\log T}$. 

% \begin{comment}
% use the Jensen inequality and the Cauchy-Schwarz inequality to obtain
% \begin{align*}
% \bigg|\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi^{\star}}\sqrt{\bm{\varphi}_{t}}\bigg|^{2} & \le\frac{1}{1-\gamma}\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi^{\star}}\bm{\varphi}_{t}
% \end{align*}
% Applying a similar argument for (\ref{eq:inf-alpha-claim}), one can
% verify that
% \[
% \sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi^{\star}}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})\lesssim\frac{1}{(1-\gamma)^{2}}\Big(1+\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\big).
% \]
% We can then bound similarly as in
% \end{comment}


 \subsection{Solving the recurrence relation regarding $\bm{\Delta}_{t}$}

\label{sec:combine_ub_lb}

 Recall from \eqref{eq:Delta-t-inf-recursion-inf-outline} that with probability exceeding $1-2\delta$, 
 the following recurrence relation  
 %
 \begin{align}
 \|\bm{\Delta}_{t}\|_{\infty} & \leq 30 \sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\bigg(1+\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\bigg)}\qquad\text{for all}\qquad t\ge\frac{T}{c_{2}\log T}\label{eq:Delta-t-inf-recursion-inf}
 \end{align}
 %
 holds, which plays a crucial role
 in establishing the desired estimation error bound. Specifically,
 for any $k\ge0$, let us define 
 \begin{equation}
 u_{k}:=\max\left\{ \|\bm{\Delta}_{t}\|_{\infty}\ \Big|\ 2^{k}\frac{T}{c_{2}\log T}\leq t\leq T\right\} .\label{eq:uk-definition-inf}
 \end{equation}
 To bound this sequence, we first obtain a crude bound as a result
 of \eqref{eq:Delta-t-range-infinite}: 
 %
 \begin{equation}
	 u_{0}\le\frac{1}{1-\gamma}. \label{eq:u0-trivial-UB}
 \end{equation}
 %
 Next, it is directly seen from \eqref{eq:Delta-t-inf-recursion-inf}
 and the definition of $u_{k}$ that
 \begin{equation}
 u_{k}\leq c_6\sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2 (1-\gamma)^{4}T}\left(1+u_{k-1}\right)},\qquad k\geq1\label{eq:uk-recursion-inf}
 \end{equation}
 for some constant $c_{6}=20/\gamma>0$. In order to analyze the size of $u_{k}$,
 we divide into two cases. 
 \begin{itemize}
 \item If $u_{k}\leq1$ for some $k\geq1$, then \eqref{eq:uk-recursion-inf}
 tells us that
 \[
 u_{k+1}\leq c_{6}\sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\left(1+u_{k}\right)}\leq c_{6}\sqrt{\frac{2\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}}\leq1,
 \]
 as long as $T\geq\frac{2c_{6}^{2}\log^{4}T\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}}{\gamma^2(1-\gamma)^{4}}$.
 In other words, once $u_{k-1}$ drops below $1$, then all subsequent
 quantities will remain bounded above by 1, namely, $\max_{j:j\geq k}u_{j}\leq1$.
 As a result, 
  %
 \[
 u_{j}\leq c_{6}\sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\left(1+u_{j-1}\right)}
		 \leq c_{6}\sqrt{\frac{2\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}}\qquad\text{for all }j>k.
 \]

 \item Instead, suppose that $u_{j}>1$ for all $0\leq j\leq k$. Then it
 is seen from \eqref{eq:uk-recursion-inf} that
 \[
 u_{j+1}\leq c_{6}\sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\left(1+u_{j}\right)}\leq c_{6}\sqrt{\frac{2\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}u_{j}}\qquad\text{for all }j\leq k.
 \]
 This is equivalent to saying that
 \[
 \log u_{j+1}\leq\log\alpha_{u}+\frac{1}{2}\log u_{j}\qquad\text{for all }j\leq k,
 \]
 where $\alpha_{u}=c_{6}\sqrt{\frac{2\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}}.$
 Invoking a standard analysis strategy for this type of recursive relations
 yields
 \[
	 \log u_{j+1} - 2\log\alpha_{u} \leq  \frac{1}{2} \left(\log u_{j}-2\log\alpha_{u}\right)\qquad\text{for all }j\leq k,
 \]
 and hence
 \[
 \log u_{j+1}\leq2\log\alpha_{u}+\left(\frac{1}{2}\right)^{j+1}\left(\log u_{0}-2\log\alpha_{u}\right)\qquad\text{for all }j\leq k.
 \]
 This is equivalent to saying that
 \[
 u_{j}\leq\alpha_{u}^{2}\left(\frac{u_{0}}{\alpha_{u}^{2}}\right)^{1/2^{j}}=\left(\alpha_{u}^{2}\right)^{1-1/2^{j}}\left(u_{0}\right)^{1/2^{j}}\qquad\text{for all }j\leq k+1.
 \]
 \end{itemize}
 %
 Putting the above two cases together and using \eqref{eq:u0-trivial-UB}, we conclude that
 %
 \begin{align*}
 u_{k} & \leq\sqrt{\frac{2c_{6}^{2}\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}}+\Big(\frac{2c_{6}^{2}\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\Big)^{1-1/2^{k}}u_{0}^{1/2^{k}}\\
  & \leq\sqrt{\frac{2c_{6}^{2}\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}}+\Big(\frac{2c_{6}^{2}\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\Big)^{1-1/2^{k}}\left(\frac{1}{1-\gamma}\right)^{1/2^{k}},\qquad k\geq1.
 \end{align*}
 In particular, as long as $k\geq c_7 \log\log\frac{1}{1-\gamma}$ for some constant $c_7>0$, one
 has $\big(\frac{1}{1-\gamma}\big)^{1/2^{k}} \leq O(1)$ and
 \[
 \Big(\frac{2c_{6}^{2}\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{4}T}\Big)^{1-1/2^{k}}\le\max\left\{ \sqrt{\frac{2c_{6}^{2}\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}},\frac{2c_{6}^{2}\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\right\} .
 \]
 As a result, the above bound simplifies to
 \begin{align*}
	 u_{k} & \leq c_8 \bigg( \sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}}+\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\bigg),
	 \qquad k\geq c_7 \log\log\frac{1}{1-\gamma} 
 \end{align*}
 %
 for some  constant $c_8>0$. 

 Consequently, taking $t=T$ and choosing $k= c_7\log\log\frac{1}{1-\gamma}$
 for some appropriate constant $c_7>0$ (so as to ensure $2^{k}\frac{T}{c_{2}\log T}<T$),
 we immediately see from the definition \eqref{eq:uk-definition-inf}
 of $u_{k}$ that 
 %
 \begin{equation}
	 \|\bm{\Delta}_{T}\|_{\infty}\leq c_8 \bigg( \sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}}+\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T} \bigg) .\label{eq:Delta-T-UB-final-inf}
 \end{equation}
 %
 with probability at least $1-2\delta$. To finish up, we note that
 the sample size assumption (\ref{eq:sample-size-condition-complete-inf})
 is equivalent to
 \[
 \frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\le\frac{\min\{\varepsilon^{2},\varepsilon\}}{c_{3}}.
 \]
 When $c_{3}>0$ is sufficiently large, substituting this relation
 into \eqref{eq:Delta-T-UB-final-inf} gives
 \begin{align*}
 \|\bm{\Delta}_{T}\|_{\infty} & \leq\frac{1}{2}\sqrt{\min\{\varepsilon^{2},\varepsilon\}}+\frac{1}{2}\min\{\varepsilon^{2},\varepsilon\}=\begin{cases}
 \frac{1}{2}\varepsilon+\frac{1}{2}\varepsilon^{2} & \text{if }\varepsilon\leq1\\
 \frac{1}{2}\sqrt{\varepsilon}+\frac{1}{2}\varepsilon\quad & \text{if }\varepsilon>1
 \end{cases}\\
  & \leq\varepsilon
 \end{align*}
 as claimed in Theorem \ref{thm:infinite-horizon}. 

\subsection{Proof of Lemma \ref{lemma:inf-alpha-claim}}

\label{subsec:Proof-of-eq:inf-alpha-claim}

We first claim that
\begin{equation}
\max_{\frac{t}{2}\le i<t}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})-\mathsf{Var}_{\bm{P}}(\bm{V}^{\star})\le\frac{4}{1-\gamma}\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\bm{1}.\label{eq:Var-P-Var-star-gap-inf}
\end{equation}
If this claim were valid (which we shall justify towards the end of
this subsection), then it would lead to
\begin{equation}
\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\max_{\frac{t}{2}\le i<t}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})\le\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\mathsf{Var}_{\bm{P}}(\bm{V}^{\star})+\frac{4}{(1-\gamma)^{2}}\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\bm{1}.\label{eq:mixture-gamma-P-Var-V-inf}
\end{equation}

It then boils down to bounding the first term on the right-hand side
of \eqref{eq:mixture-gamma-P-Var-V-inf}. 
%Towards this, let us first define a subvector of $\bm{Q}^{\star}$ as follows
%%
%\begin{equation}
%\bm{Q}_{\pi}^{\star}:=\bm{r}_{\pi} +\gamma\bm{P}_{\pi}\bm{V}^{\star}\in\mathbb{R}^{|\mathcal{S}|},\label{def:Q-sub-pi}
%\end{equation}
%%
%where we denote by 
%$\bm{r}_{\pi}:=\bm{\Pi}^{\pi}\bm{r}\in\mathbb{R}^{|\mathcal{S}|}$
%the reward vector restricted to the actions chosen by the policy $\pi$,
%namely, $r_{\pi}(s)=r(s,\pi(s))$ for all $s\in\mathcal{S}$. 
Let us first upper bound the variance term involving $\bm{V}^{\star}$.
For any $0\leq h<H$, one can express (see \eqref{eq:defn-Var-P-V})
%
\begin{align}
\mathsf{Var}_{\bm{P}}(\bm{V}^{\star}) & =\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-(\bm{P}\bm{V}^{\star})\circ(\bm{P}\bm{V}^{\star})\nonumber\\
 & \overset{(\mathrm{i})}{=}\bm{P}^{\pi_{i_{h+1}}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})+\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-\bm{P}^{\pi_{i_{h+1}}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})-\frac{1}{\gamma^{2}}(\bm{Q}^{\star}-\bm{r})\circ(\bm{Q}^{\star}-\bm{r})\nonumber\\
 & =\bm{P}^{\pi_{i_{h+1}}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})+\bm{P}^{\pi^{\star}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})-\bm{P}^{\pi_{i_{h+1}}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})-\frac{1}{\gamma^{2}}(\bm{Q}^{\star}-\bm{r})\circ(\bm{Q}^{\star}-\bm{r})\nonumber\\
 & \leq\bm{P}^{\pi_{i_{h+1}}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})+\big\|\bm{P}^{\pi^{\star}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})-\bm{P}^{\pi_{i_{h+1}}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})\big\|_{\infty}\bm{1}-\frac{1}{\gamma^{2}}(\bm{Q}^{\star}-\bm{r})\circ(\bm{Q}^{\star}-\bm{r})\nonumber\\
 & \overset{(\mathrm{ii})}{\leq}\bm{P}^{\pi_{i_{h+1}}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})+\frac{4}{1-\gamma}\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\bm{1}-\frac{1}{\gamma^{2}}(\bm{Q}^{\star}-\bm{r})\circ(\bm{Q}^{\star}-\bm{r})\nonumber\\
 & =\frac{1}{\gamma^{2}}\big(\gamma^{2}\bm{P}^{\pi_{i_{h+1}}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})-\bm{Q}^{\star}\circ\bm{Q}^{\star}\big)+\frac{4}{1-\gamma}\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\bm{1}-\frac{1}{\gamma^{2}}\bm{r}\circ\bm{r}+\frac{2}{\gamma^{2}}\bm{Q}^{\star}\circ\bm{r}\nonumber\\
 & \overset{(\mathrm{iii})}{\leq}\frac{1}{\gamma}\big(\gamma\bm{P}^{\pi_{i_{h+1}}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})-\bm{Q}^{\star}\circ\bm{Q}^{\star}\big)+\frac{2}{\gamma^{2}}\bm{Q}^{\star}\circ\bm{r}+\frac{4}{1-\gamma}\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\bm{1},\label{eq:var-V-UB}
\end{align}
%
where (i) relies on the identity $\bm{Q}^{\star}=\bm{r}+\gamma\bm{P}\bm{V}^{\star}$,
and (iii) holds since $0<\gamma<1$. To justify (ii), we make the
following observation:
%
\begin{align*}
 & \big\|\bm{P}^{\pi_{i_{h+1}}}\big(\bm{Q}^{\star}\circ\bm{Q}^{\star}\big)-\bm{P}^{\pi^{\star}}\big(\bm{Q}^{\star}\circ\bm{Q}^{\star}\big)\big\|_{\infty}=\big\|\bm{P}\bm{\Pi}^{\pi_{i_{h+1}}}\big(\bm{Q}^{\star}\circ\bm{Q}^{\star}\big)-\bm{P}\bm{\Pi}^{\pi^{\star}}\big(\bm{Q}^{\star}\circ\bm{Q}^{\star}\big)\big\|_{\infty}\\
% & \quad\leq\|\bm{P}\|_{1}\big\|\bm{\Pi}^{\pi_{i_{h+1}}}\big(\bm{Q}^{\star}\circ\bm{Q}^{\star}\big)-\bm{\Pi}^{\pi^{\star}}\big(\bm{Q}^{\star}\circ\bm{Q}^{\star}\big)\big\|_{\infty}\\
 & \quad\overset{(\mathrm{iv})}{\leq}\big\|\bm{\Pi}^{\pi_{i_{h+1}}}\big(\bm{Q}^{\star}\circ\bm{Q}^{\star}\big)-\bm{\Pi}^{\pi^{\star}}\big(\bm{Q}^{\star}\circ\bm{Q}^{\star}\big)\big\|_{\infty}\\
 & \quad=\big\|\big(\bm{\Pi}^{\pi_{i_{h+1}}}\bm{Q}^{\star}-\bm{\Pi}^{\pi^{\star}}\bm{Q}^{\star}\big)\circ\big(\bm{\Pi}^{\pi_{i_{h+1}}}\bm{Q}^{\star}+\bm{\Pi}^{\pi^{\star}}\bm{Q}^{\star}\big)\big\|_{\infty}\\
 & \quad\overset{(\mathrm{v})}{\leq}\frac{2}{1-\gamma}\big\|\bm{\Pi}^{\pi_{i_{h+1}}}\bm{Q}^{\star}-\bm{\Pi}^{\pi^{\star}}\bm{Q}^{\star}\big\|_{\infty}\\
 & \quad\le\frac{2}{1-\gamma}\Big(\big\|\bm{\Pi}^{\pi_{i_{h+1}}}\bm{Q}^{\star}-\bm{\Pi}^{\pi_{i_{h+1}}}\bm{Q}_{i_{h+1}}\big\|_{\infty}+\big\|\bm{\Pi}^{\pi_{i_{h+1}}}\bm{Q}_{i_{h+1}}-\bm{V}^{\star}\big\|_{\infty}\Big)\\
	& \quad \overset{\mathrm{(vi)}}{\leq} \frac{2}{1-\gamma}\Big(\big\|\bm{Q}^{\star}-\bm{Q}_{i_{h+1}}\big\|_{\infty}+\big\|\bm{V}_{i_{h+1}}-\bm{V}^{\star}\big\|_{\infty}\Big)\\
 & \quad\overset{\mathrm{(vii)}}{\leq}\frac{4}{1-\gamma}\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty},
\end{align*}
%
where (iv) arises from the fact $\|\bm{P}\bm{z}\|_{\infty}\leq\|\bm{P}\|_{1}\|\bm{z}\|_{\infty}=\|\bm{z}\|_{\infty}$, 
(v) is valid because $\|\bm{Q}^{\star}\|_{\infty}\leq  1/(1-\gamma)$,  (vi) follows from the fact that $\bm{V}_{i_{h+1}}=\bm{\Pi}^{\pi_{i_{h+1}}}\bm{Q}_{i_{h+1}}$,
and (vi) holds since $\|\bm{V}_{i_{h+1}}-\bm{V}^{\star}\|_{\infty}\leq \|\bm{Q}_{i_{h+1}}-\bm{Q}^{\star}\|_{\infty}$.  
%
%\begin{align*}
% & \big\|\bm{P}^{\pi_{i_{h+1}}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})-\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})\big\|_{\infty}
% =\big\|\bm{P}\big(\bm{Q}_{\pi_{i_{h+1}}}^{\star}\circ\bm{Q}_{\pi_{i_{h+1}}}^{\star}-\bm{V}^{\star}\circ\bm{V}^{\star}\big)\big\|_{\infty}\\
% & \quad\le\big\|\bm{Q}_{\pi_{i_{h+1}}}^{\star}\circ\bm{Q}_{\pi_{i_{h+1}}}^{\star}-\bm{V}^{\star}\circ\bm{V}^{\star}\big\|_{\infty}
% =\big\|\big(\bm{Q}_{\pi_{i_{h+1}}}^{\star}-\bm{V}^{\star}\big)\circ\big(\bm{Q}_{\pi_{i_{h+1}}}^{\star}+\bm{V}^{\star} \big)\big\|_{\infty}\\
% & \quad\le\frac{2}{1-\gamma}\big\|\bm{Q}_{\pi_{i_{h+1}}}^{\star}-\bm{V}^{\star}\big\|_{\infty}\le\frac{2}{1-\gamma}\Big(\big\|\bm{Q}_{\pi_{i_{h+1}}}^{\star}-\bm{V}_{i_{h+1}}\big\|_{\infty}+\big\|\bm{V}_{i_{h+1}}-\bm{V}^{\star}\big\|_{\infty}\Big)\\
% & \quad\le\frac{4}{1-\gamma}\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty},
%\end{align*}
%%
%where the first inequality arises from the fact $\|\bm{P}\bm{z}\|_{\infty}\leq\|\bm{P}\|_{1}\|\bm{z}\|_{\infty}=\|\bm{z}\|_{\infty}$,
%the second inequality follows from Lemma \ref{lemma:non-negativity-Qt-Vt},
%and the last line holds since
%%
%\begin{align*}
%\big\|\bm{Q}_{\pi_{i_{h+1}}}^{\star}-\bm{V}_{i_{h+1}}\big\|_{\infty} & =\big\|\bm{\Pi}^{\pi_{i_{h+1}}}\bm{Q}^{\star}-\bm{\Pi}^{\pi_{i_{h+1}}}\bm{Q}_{i_{h+1}}\big\|_{\infty}\leq\big\|\bm{Q}^{\star}-\bm{Q}_{i_{h+1}}\big\|_{\infty}=\|\bm{\Delta}_{i_{h+1}}\|_{\infty} \\
%%
%&\text{and} \qquad
%\|\bm{V}_{i_{h+1}}-\bm{V}^{\star}\|_{\infty}  \le\big\|\bm{Q}_{i_{h+1}} - \bm{Q}^{\star}\big\|_{\infty}= \|\bm{\Delta}_{i_{h+1}}\|_{\infty}.
%\end{align*}

As it turns out, the first term in \eqref{eq:var-V-UB} allows one to build a telescoping sum. Specifically, invoking (\ref{eq:var-V-UB}) allows one to bound
%
\begin{align}
\sum_{h=0}^{H-1}\prod_{k=1}^{h}\gamma\bm{P}^{\pi_{i_{k}}}\mathsf{Var}_{\bm{P}}(\bm{V}^{\star}) & \leq\frac{1}{\gamma}\sum_{h=0}^{H-1}\prod_{k=1}^{h}\gamma\bm{P}^{\pi_{i_{k}}}\big(\gamma\bm{P}^{\pi_{i_{h+1}}}(\bm{Q}^{\star}\circ\bm{Q}^{\star})-\bm{Q}^{\star}\circ\bm{Q}^{\star}\big) \nonumber \\
 & \qquad\qquad+\frac{4}{1-\gamma}\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\bm{1}+\frac{2}{\gamma^{2}}\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\left(\bm{Q}^{\star}\circ\bm{r}\right) \nonumber \\
 & \overset{(\mathrm{i})}{=}\frac{1}{\gamma}\Big(\sum_{h=0}^{H-1}\prod_{k=1}^{h+1}\gamma\bm{P}^{\pi_{i_{k}}}-\sum_{h=0}^{H-1}\prod_{k=1}^{h}\gamma\bm{P}^{\pi_{i_{k}}}\Big)\left(\bm{Q}^{\star}\circ\bm{Q}^{\star}\right) \nonumber \\
 & \qquad\qquad+\frac{4}{1-\gamma}\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\sum_{h=0}^{H-1}\gamma^{h}\bm{1}+\frac{2}{\gamma^{2}}\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\left(\bm{Q}^{\star}\circ\bm{r}\right) \nonumber \\
 & \leq\frac{1}{\gamma}\bigg(\prod_{k=1}^{H}\gamma\bm{P}^{\pi_{i_{k}}}-\bm{I}\bigg)\left(\bm{Q}^{\star}\circ\bm{Q}^{\star}\right)+\frac{4}{(1-\gamma)^{2}}\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\bm{1} \nonumber \\
 & \qquad\qquad+\frac{2}{\gamma^{2}}\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\left(\bm{Q}^{\star}\circ\bm{r}\right) \nonumber \\
 & \overset{(\mathrm{ii})}{\leq}\bigg(\frac{2}{\gamma}\|\bm{Q}^{\star}\|_{\infty}^{2}+\frac{4}{(1-\gamma)^{2}}\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}+\frac{2}{\gamma^{2}}\frac{1}{1-\gamma}\|\bm{Q}^{\star}\|_{\infty}\|\bm{r}\|_{\infty}\bigg)\bm{1} \nonumber \\
 & \overset{(\mathrm{iii})}{\leq}\frac{1}{(1-\gamma)^{2}}\bigg(\frac{2}{\gamma}+4\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}+\frac{2}{\gamma^{2}}\bigg)\bm{1} \nonumber \\
 & \le\frac{1}{(1-\gamma)^{2}}\bigg(\frac{4}{\gamma^{2}}+4\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\bigg)\bm{1}. \label{eq:sum-prod-P-var-UB}
\end{align}
Here, (i) comes from the identity $\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}\bm{1}=\bm{1}$;
(ii) holds because each row of $\prod_{k=1}^{h}\bm{P}^{\pi_{i_{k}}}$
has unit $\|\cdot\|_{1}$ norm for any $h$; (iii) arises from the
bound $\|\bm{Q}^{\star}\|_{\infty}\leq1/(1-\gamma)$. This completes
the proof, as long as the claim \eqref{eq:Var-P-Var-star-gap-inf}
can be justified.

\paragraph{Proof of the inequality \eqref{eq:Var-P-Var-star-gap-inf}.}
%
To validate this result, we make the observation that
\begin{align*}
\mathsf{Var}_{\bm{P}}(\bm{V}_{i})-\mathsf{Var}_{\bm{P}}(\bm{V}^{\star}) & =\big[\bm{P}(\bm{V}_{i}\circ\bm{V}_{i})-(\bm{P}\bm{V}_{i})\circ(\bm{P}\bm{V}_{i})\big]-\big[\bm{P}(\bm{V}^{\star}\circ\bm{V}^{\star})-(\bm{P}\bm{V}^{\star})\circ(\bm{P}\bm{V}^{\star})\big]\\
 & =\bm{P}(\bm{V}_{i}\circ\bm{V}_{i}-\bm{V}^{\star}\circ\bm{V}^{\star})+(\bm{P}\bm{V}^{\star})\circ(\bm{P}\bm{V}^{\star})-(\bm{P}\bm{V}_{i})\circ(\bm{P}\bm{V}_{i})\\
	& =\bm{P}\big((\bm{V}_{i}-\bm{V}^{\star})\circ(\bm{V}_{i}+\bm{V}^{\star}) \big)
	+(\bm{P}\bm{V}^{\star}-\bm{P}\bm{V}_{i})\circ(\bm{P}\bm{V}^{\star}+\bm{P}\bm{V}_{i})\\
	& \le\Big\{ \big\|\bm{P}\big( (\bm{V}_{i}-\bm{V}^{\star}) \circ(\bm{V}_{i}+\bm{V}^{\star}) \big)\big\|_{\infty}
	+ \big\|(\bm{P}\bm{V}^{\star}-\bm{P}\bm{V}_{i})\circ(\bm{P}\bm{V}^{\star}+\bm{P}\bm{V}_{i}) \big\|_{\infty}\Big\}\bm{1}\\
 & \leq\frac{4}{1-\gamma}\|\bm{\Delta}_{i}\|_{\infty}\bm{1}.
\end{align*}
Here, the last inequality follows since (by applying Lemma \ref{lemma:non-negativity-Qt-Vt})
\[
	\big\|\bm{P}\big( (\bm{V}_{i}-\bm{V}^{\star})\circ(\bm{V}_{i}+\bm{V}^{\star}) \big)\big\|_{\infty}\le\|\bm{P}\|_{1}\|\bm{V}_{i}-\bm{V}^{\star}\|_{\infty}\|\bm{V}_{i}+\bm{V}^{\star}\|_{\infty}\le\frac{2}{1-\gamma}\|\bm{\Delta}_{i}\|_{\infty},
\]
\[
\text{and}\quad 
\big\|(\bm{P}\bm{V}^{\star}-\bm{P}\bm{V}_{i})\circ(\bm{P}\bm{V}^{\star}+\bm{P}\bm{V}_{i}) \big\|_{\infty}\le\|\bm{P}\|_{1}\|\bm{V}_{i}-\bm{V}^{\star}\|_{\infty}\cdot\|\bm{P}\|_{1}\|\bm{V}_{i}+\bm{V}^{\star}\|_{\infty}\le\frac{2}{1-\gamma}\|\bm{\Delta}_{i}\|_{\infty}.
\]


\paragraph{A useful extension of Lemma~\ref{lemma:inf-alpha-claim}.}
Before concluding, we  make note of the following extension that proves useful for studying asynchronous Q-learning. 
%
\begin{lemma}
\label{lemma:inf-alpha-claim-async}
Suppose that $t\geq \frac{T}{c_2\log T}$. Then one has
%
\begin{equation}
	\sum_{h=0}^{H-1}\gamma^{h}\prod_{k=1}^{h}\bm{P}^{\widehat{\pi}_{k}}\max_{\frac{t}{2}\le i<t}\mathsf{Var}_{\bm{P}}(\bm{V}_{i})
	\leq \frac{4}{\gamma^2(1-\gamma)^{2}}\Big(1+2\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\Big)\bm{1} \label{eq:inf-alpha-claim}
\end{equation}
%
for any set of policies $\{\widehat{\pi}_{k}\}$ obeying $\{\widehat{\pi}_{k}\}\subseteq \Pi$.  Here, we define
%
\begin{eqnarray}
\label{defn:Pi-asyncQ-123}
\begin{aligned}
	\Pi  \defn\big\{\pi=[\pi(s)]_{s\in\cS}\, \big| \, \pi(s) \in\Pi_{s}, \forall s\in\cS\big\}, 
	\qquad \Pi_{s}  \defn\big\{ \pi_{i}(s) \mid i\in[t/2,t)\big\}. 
\end{aligned}
\end{eqnarray}
%
\end{lemma}
%
The key difference between Lemma~\ref{lemma:inf-alpha-claim-async} and Lemma~\ref{lemma:inf-alpha-claim} is that: 
the components of $\widehat{\pi}_{k}$ corresponding to different states can be chosen in a separate manner. 
The proof follows from an identical argument as the above proof of Lemma~\ref{lemma:inf-alpha-claim}, and is hence omitted. 


% \begin{comment}
% \begin{theorem}\label{thm:infinite-horizon-lower}Suppose that the
% step size and the iteration number satisfy
% \[
% \eta_{t}=\frac{1}{1+c_{1}t}\qquad\text{or}\qquad\eta_{t}\equiv c_{2}.
% \]
% Then for any $0<\varepsilon<1$ and $T\le\frac{1}{(1-\gamma)^{4}\varepsilon^{2}}$,
% and any parameters $c_{1},c_{2}>0$, there exists some MDP, such that
% for some state $s\in\mathcal{S}$, 
% \[
% \mathbb{E}\big|V_{T}(s)-V^{\star}(s)\big|^{2}\ge\varepsilon^{2}.
% \]
% \end{theorem}

% We consider the following MDP $(\mathcal{S},\mathcal{A},\bm{P},\bm{r},\gamma)$,
% where $\mathcal{S}=\{0,1\}$, $\mathcal{A}=\{0\}$, 
% \[
% P(0\mymid0,0)=1,P(1\mymid1,0)=p,P(0\mymid1,0)=1-p,\qquad\text{for}\qquad p=\gamma+\frac{2\gamma(1-\gamma)^{2}\varepsilon}{(1+\gamma)^{2}},
% \]
%  and 
% \[
% r(0,0)=r,r(1,0)=1,
% \]
% with $r=0$ if $V_{0}(0)>\frac{1}{4(1-\gamma)}$ and $r=0.5$ otherwise.
% Then it can be seen that
% \[
% V^{\star}(0)=\frac{r}{1-\gamma},\qquad\text{and}\qquad V^{\star}(1)=.
% \]
% Here, the update rule (\ref{eqn:q-learning}) can be rewritten
% as
% \begin{align*}
% V_{t}(0) & =(1-\eta_{t})V_{t-1}(0)+\eta_{t}(r+\gamma V_{t-1}(0)),\\
% V_{t}(1) & =(1-\eta_{t})V_{t-1}(1)+\eta_{t}(1+\gamma\bm{P}_{t}\bm{V}_{t-1}),
% \end{align*}
% where $\bm{P}_{t}$ denotes . Then it can be seen from the above expression
% \begin{align*}
% V_{t}(0) & =(1-(1-\gamma)\eta_{t})V_{t-1}(0)+\eta_{t}r\\
%  & =\prod_{i=1}^{t}(1-(1-\gamma)\eta_{i})V_{0}(0)+\sum_{k=1}^{t}\eta_{k}\prod_{i=k+1}^{t}(1-(1-\gamma)\eta_{i})r\\
%  & =\prod_{i=1}^{t}(1-(1-\gamma)\eta_{i})\Big(V_{0}(0)-\frac{r}{1-\gamma}\Big)+\frac{r}{1-\gamma},
% \end{align*}
% where the last line comes from the identity 
% \[
% \prod_{i=1}^{t}(1-(1-\gamma)\eta_{i})+\sum_{k=1}^{t}(1-\gamma)\eta_{k}\prod_{i=k+1}^{t}(1-(1-\gamma)\eta_{i})=1.
% \]
% Hence, if $c_{1}>\frac{1-\gamma}{\log T}$ or $c_{2}<\frac{1}{(1-\gamma)T}$,
% we have by calculation
% \begin{align*}
% \big|V_{t}(0)-V^{\star}(0)\big| & =\prod_{i=1}^{t}(1-(1-\gamma)\eta_{i})\Big|V_{0}(0)-\frac{r}{1-\gamma}\Big|\\
%  & \ge\frac{1}{4e(1-\gamma)},
% \end{align*}
% which establishes the claim.

% Then assume that $c_{1}\le\frac{1-\gamma}{\log T}$ or $c_{2}\ge\frac{1}{(1-\gamma)T}$.
% In addition, if 
% \[
% \prod_{i=1}^{T/2}(1-(1-\gamma)\eta_{i})>\sqrt{\varepsilon(1-\gamma)}
% \]
% Hence, we can also assume that $\big|V_{t}(0)-V^{\star}(0)\big|\le\sqrt{\frac{\varepsilon}{1-\gamma}}$
% for $t\ge\frac{T}{2}$.
% \[
% \bm{\Delta}_{t}=(1-\eta_{t})\bm{\Delta}_{t-1}+\gamma\eta_{t}\big[\bm{P}\bm{\Delta}_{t-1}+(\bm{P}_{t}-\bm{P})\bm{V}_{t-1}\big],
% \]
% \[
% \bm{\Delta}_{t}=\prod_{i=1}^{t}\big[\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big]\bm{\Delta}_{0}+\gamma\sum_{k=1}^{t}\eta_{k}\prod_{i=k+1}^{t}\big[\bm{I}-\eta_{i}(\bm{I}-\gamma\bm{P})\big](\bm{P}_{k}-\bm{P})\bm{V}_{k-1},
% \]
% \end{comment}


introduction.tex
\section{Introduction}

Q-learning is arguably one of the most widely adopted model-free algorithms \citep{watkins1989learning,watkins1992q}. 
Characterizing its sample efficiency  lies at the core of the statistical foundation of reinforcement learning (RL) \citep{sutton2018reinforcement}.
While classical convergence analyses for Q-learning \citep{tsitsiklis1994asynchronous,szepesvari1998asymptotic,jaakkola1994convergence,borkar2000ode} focused primarily on the asymptotic regime --- in which the number of iterations tends to infinity with other problem parameters held fixed --- recent years have witnessed a paradigm shift from asymptotic analyses towards a finite-sample\,/\,finite-time framework \citep{even2003learning,kearns1999finite,beck2012error,chen2020finite,wainwright2019stochastic,lee2018stochastic,qu2020finite,xiong2020finitetime,chen2021lyapunov,li2020sample,weng2020momentum}.  Drawing insights from high-dimensional statistics \citep{wainwright2019high},  a modern non-asymptotic framework unveils more clear and informative impacts of salient problem parameters upon the sample complexity, particularly for those applications with enormous state/action space and long horizon. 
Motivated by its practical value, a suite of non-asymptotic theory has been recently developed for Q-learning to accommodate multiple sampling mechanisms \citep{even2003learning,beck2012error,jin2018q,wainwright2019stochastic,qu2020finite,li2020sample}.


%and seeks to characterize the dependency on salient problem parameters in an non-asymptotic fashion, leads 
%
%
%n explosion of activities towards developing finite-sample or finite-time performance guarantees of Q-learning, e.g.~\citet{even2003learning,kearns1999finite,beck2012error,chen2020finite,wainwright2019stochastic,qu2020finite,xiong2020finitetime,li2020sample}. 
%
% in learning the optimal policy in a tabular Markov decision process (MDP) under different data collection mechanisms
 

In this paper, we revisit the sample complexity of Q-learning for tabular Markov decision processes (MDPs).
For concreteness, let us consider the synchronous setting, which assumes access to a generative model or a simulator that produces independent samples for all state-action pairs  in each iteration \citep{kearns2002sparse,kakade2003sample}; 
this setting is termed ``synchronous'' as the estimates w.r.t.~all state-action pairs are updated at once. We investigate the $\ell_{\infty}$-based sample complexity, namely, the number of samples needed for synchronous Q-learning to yield an entrywise $\varepsilon$-accurate estimate of the optimal Q-function. Despite a number of prior works tackling this setting, the dependence of the sample complexity on the effective horizon $\frac{1}{1-\gamma}$ remains unsettled. Take $\gamma$-discounted infinite-horizon MDPs for instance: the state-of-the-art sample complexity bounds \citep{wainwright2019stochastic,chen2020finite} scale on the order of $ \frac{|\cS||\cA|}{(1-\gamma)^5\varepsilon^{2}} $ (up to some log factor), where $\cS$ and $\cA$ represent the state space and the action space, respectively. However, it is unclear whether this scaling is sharp for Q-learning,  and whether it can be further improved via a more refined theory. On the one hand, the minimax lower limit for this setting has been shown to be on the order of $ \frac{|\cS||\cA|}{(1-\gamma)^3\varepsilon^{2}} $ (up to some log factor) \citep{azar2013minimax}; this limit is achievable by model-based approaches \citep{agarwal2019optimality,li2020breaking} and apparently smaller than prior sample complexity bounds for Q-learning. On the other hand, \citet{wainwright2019variance} argued through numerical experiments that ``{\em the usual Q-learning suffers from at least worst-case fourth-order scaling in the discount complexity $\frac{1}{1-\gamma}$, as opposed
to the third-order scaling $\ldots$}'', although no rigorous justification was provided therein.   
Given the gap between the achievability bounds and lower bounds in the status quo, it is natural to seek answers to the following questions:   
%
\begin{center}
{\em What is the tight sample complexity characterization of Q-learning?} \\
{\em How does it compare to the minimax sample complexity limit? }
\end{center}
%
%\yjc{maybe we can remove the original question}


%exhibited a numerical example that hints at the non-sharpness of existing Q-learning theory; more specifically, the numerical experiments conducted therein suggested a scaling of $ \frac{|\cS||\cA|}{(1-\gamma)^4\varepsilon^{2}} $ for certain MDP instances,

% about the sample efficiency of Q-learning in comparison to its model-based counterpart, and calls for further analysis to bridge this gap. 




%known to date
%
%
%
%
%



\newcommand{\topsepremove}{\aboverulesep = 0mm \belowrulesep = 0mm} \topsepremove



\begin{table*}[t]
\centering
\begin{tabular}{c|c|c}
\toprule 
paper $\vphantom{\frac{1^{7}}{1^{7^3}}}$ &  learning rates & sample complexity   \\ \toprule
	\cite{even2003learning} $\vphantom{\frac{1^{7^{7^7}}}{1^{7^{7^{7^7}}}}}$ & linear: $\frac{1}{t}$ & $ 2^{\frac{1}{1-\gamma}}\frac{|\cS||\cA|}{(1-\gamma)^4\varepsilon^2}$    \\ \hline
	\cite{even2003learning} $\vphantom{\frac{1^{7^{7^7}}}{1^{7^{7^{7^7}}}}}$  & polynomial: $\frac{1}{t^{\omega}}$,\, $\omega \in (1/2,1)$ & $|\cS||\cA| \Big\{ \big( \frac{1}{(1-\gamma)^4\varepsilon^2}\big)^{1/\omega} +  \big( \frac{1}{1-\gamma }\big)^{\frac{1}{1-\omega}} \Big\}$   \\ \hline
\cite{beck2012error} $\vphantom{\frac{1^{7^{7^7}}}{1^{7^{7^{7^7}}}}}$  & constant: $\frac{(1-\gamma)^4\varepsilon^2}{|\cS||\cA|}$ & $\frac{|\cS|^2|\cA|^2}{(1-\gamma)^5\varepsilon^2} $ \\ \hline
\cite{wainwright2019stochastic} $\vphantom{\frac{1^{7^{7^7}}}{1^{7^{7^{7^7}}}}}$ & rescaled linear: $\frac{1}{1+(1-\gamma)t}$ & $\frac{|\cS||\cA|}{(1-\gamma)^5\varepsilon^2} $ \\ \hline
\cite{wainwright2019stochastic} $\vphantom{\frac{1^{7^{7^7}}}{1^{7^{7^{7^7}}}}}$  & polynomial:  $\frac{1}{t^{\omega}}$, $\omega \in (0,1)$ &   $|\cS||\cA| \Big\{ \big( \frac{1}{(1-\gamma)^4\varepsilon^2}\big)^{1/\omega} +  \big( \frac{1}{1-\gamma }\big)^{\frac{1}{1-\omega}} \Big\}$  \\ \hline
	\cite{chen2020finite} $\vphantom{\frac{1^{7^{7}}}{1^{7^{7^{7^{7^{7^7}}}}}}}$  & rescaled linear: $\frac{1}{\frac{1}{(1-\gamma)^2}+(1-\gamma)t}$ & $\frac{ |\cS||\cA|  }{(1-\gamma)^5\varepsilon^2}$ \\ \hline
\cite{chen2020finite} $\vphantom{\frac{1^{7^{7^7}}}{1^{7^{7^{7^7}}}}}$  & constant: $(1-\gamma)^4\varepsilon^2$ & $\frac{ |\cS||\cA|  }{(1-\gamma)^5\varepsilon^2} $ \\ \hline
	{\bf this work (Q-learning, $|\cA|\geq 2$)} $\vphantom{\frac{1^{7^{7^7}}}{1^{7^{7^{7^7}}}}}$  & rescaled linear: $\frac{1}{1+ (1-\gamma)t}$  &  $\frac{ |\cS||\cA|  }{(1-\gamma)^4\varepsilon^2} $  \\ \hline
	{\bf this work (Q-learning, $|\cA|\geq 2$)} $\vphantom{\frac{1^{7^{7^7}}}{1^{7^{7^{7^7}}}}}$  & constant: $(1-\gamma)^3 \varepsilon^2$ &  $\frac{ |\cS||\cA|  }{(1-\gamma)^4\varepsilon^2} $  \\ \toprule
	{\bf this work (TD learning, $|\cA|=1$)} $\vphantom{\frac{1^{7^{7^7}}}{1^{7^{7^{7^7}}}}}$  & rescaled linear: $\frac{1}{1+ (1-\gamma)t}$  &  $\frac{ |\cS|   }{(1-\gamma)^3\varepsilon^2} $  \\ \hline
	{\bf this work (TD learning, $|\cA|=1$)} $\vphantom{\frac{1^{7^{7^7}}}{1^{7^{7^{7^7}}}}}$  & constant: $(1-\gamma)^3 \varepsilon^2$ &  $\frac{ |\cS|  }{(1-\gamma)^3\varepsilon^2} $  \\ \toprule
	%	{\bf lower bound (this work)} $\vphantom{\frac{1^{7^{7^7}}}{1^{7^{7^{7^7}}}}}$  & constant or rescaled linear &  $\frac{ |\cS||\cA|  }{(1-\gamma)^4\varepsilon^2} $  \\ \toprule
\end{tabular}
%
	\caption{Comparisons of existing sample complexity upper bounds of {\em synchronous} Q-learning and TD learning for an infinite-horizon $\gamma$-discounted MDP with state space $\cS$ and action space $\cA$, where $0<\varepsilon<1$ is the target accuracy level. Here, sample complexity refers to the total number of samples  needed to yield either $\max_{s,a} |\widehat{Q}(s,a)- {Q}^{\star}(s,a)|\leq \varepsilon $ with high probability or $\mathbb{E}\big[\max_{s,a} |\widehat{Q}(s,a)- {Q}^{\star}(s,a)| \big] \leq \varepsilon$, where $\widehat{Q}$ is the estimate returned by Q-learning. All logarithmic factors are omitted in the table to simplify the expressions.   \label{table:comparisons}}
\end{table*}


%Note that the total sample complexity needs to be multiplied by a factor of $|\cS||\cA|$. We also consider the worst-case scaling of \cite{wainwright2019stochastic} with bounded rewards. \cite{chen2020finite} bounded $\mathbb{E}[\|\bm{Q}_t-\bm{Q}^{\star}\|_{\infty}^2]$. We translated their bound to meet $\mathbb{E}[\|\bm{Q}_t-\bm{Q}^{\star}\|_{\infty}^2]\leq \varepsilon^2$.

 


\subsection{Main contributions}

Focusing on $\gamma$-discounted infinite-horizon MDPs with state space $\cS$ and action space $\cA$, 
this paper settles the $\ell_{\infty}$-based sample complexity of synchronous Q-learning.
Here and throughout, the standard notation $f(\cdot)=\widetilde{O} ( g(\cdot) )$ (resp.~$f(\cdot)=\widetilde{\Omega} ( g(\cdot) )$) 
means that $f(\cdot)$ is orderwise no larger than (resp.~no smaller than) $g(\cdot)$ modulo some logarithmic factors. Our main contributions regarding synchronous Q-learning are summarized below.
%
\begin{itemize}

	\item  When $|\cA|=1$, Q-learning coincides with temporal difference (TD) learning in a Markov reward process. For any $0<\varepsilon<1$, we prove that a total sample size of 
		%
	      \begin{equation}
		      \widetilde{O} \Big( \frac{|\cS|}{(1-\gamma)^3\varepsilon^2} \Big)
		\end{equation}
		% 
		is sufficient for TD learning to guarantee $\varepsilon$-accuracy in an $\ell_{\infty}$ sense; see Theorem~\ref{thm:policy-evaluation}. This is sharp and minimax optimal (up to some log factor).


	\item Moving on to the case with $|\cA|\geq 2$, we demonstrate that a sample size of  
		%
	      \begin{equation}
		      \widetilde{O} \Big( \frac{|\cS||\cA|}{(1-\gamma)^4\varepsilon^2} \Big)
		\end{equation}
		%
		suffices for Q-learning to yield $\varepsilon$-accuracy in an $\ell_{\infty}$ sense for any $0<\varepsilon<1$; 
		see Theorem~\ref{thm:infinite-horizon-simple}. 
		Conversely, we construct a hard MDP instance with 4 states and 2 actions, for which Q-learning provably requires at least
		%
      		\begin{equation}
		      \widetilde{\Omega} \Big( \frac{1} {(1-\gamma)^4\varepsilon^2} \Big)
		\end{equation}
		%
		iterations to achieve $\varepsilon$-accuracy in an $\ell_{\infty}$ sense; see Theorem~\ref{thm:LB-example}. 
		These two theorems taken collectively lead to the first sharp characterization of the sample complexity of Q-learning,
		strengthening prior theory \citep{wainwright2019stochastic,chen2020finite} by a factor of $\frac{1}{1-\gamma}$. 
		In addition, the discrepancy between our sharp characterization and the minimax lower bound makes clear that Q-learning is {\em not} minimax optimal when $|\cA|\geq 2$, and is outperformed by, say, the model-based approaches \citep{agarwal2019optimality,li2020breaking} in terms of the sample efficiency. 



		% and, at the same time, reveals the sub-optimality of Q-learning when $|\cA|\geq 2$. 
		
\end{itemize}
%
Our results cover both rescaled linear and constant learning rates; see Table~\ref{table:comparisons} for more detailed comparisons with previous literature. 
On the technical side, 
(i) our analysis for the upper bound relies on a sort of crucial error decompositions and variance control that are previously unexplored, which might shed light on how to pin down the finite-sample efficacy of other variants of Q-learning such as double Q-learning; 
(ii) the development of our lower bound, which is inspired by  \cite{azar2013minimax,wainwright2019variance}, puts the negative impact of over-estimation on sample efficiency on a rigorous footing.  


Finally, we extend our analysis framework to accommodate the asynchronous setting, in which the samples are non-i.i.d.~and take the form of a single Markovian trajectory. We show for the first time that the sample complexity of asynchronous Q-learning exhibits a $\frac{1}{(1-\gamma)^4}$ scaling w.r.t.~the effective horizon, improving upon the prior state-of-the-art \cite{li2020sample}. 




% In addition, it is also worth emphasizing that our sample complexity bound matches the theory for speedy Q-learning \citep{ghavamzadeh2011speedy} without requiring extra computation and storage. 



%asynchronous Q-learning and 

% Encouragingly, our theoretical guarantees provide a decisive answer to the sample complexity of synchronous Q-learning, which is on the order of $\frac{|\cS||\cA|}{(1-\gamma)^4\varepsilon^2}$ (up to some logarithmic factor). 
% \citet{wainwright2019variance} on a rigorous footing (more specifically, ).  

%
%As a worthy note, our work also puts the conjecture in \citet{wainwright2019variance} on a rigorous footing (more specifically, \citet{wainwright2019variance} argued through numerical experiments that ``{\em the usual Q-learning suffers from at least worst-case fourth-order scaling in the discount complexity $\frac{1}{1-\gamma}$, as opposed
%to the third-order scaling $\ldots$ }'').  


 


\subsection{Related works}

There is a growing literature dedicated to analyzing the non-asymptotic behavior of value-based model-free RL algorithms in a variety of scenarios. 
In the discussion below, we subsample the literature and discuss a couple of papers that are the closest to ours. 

% under various function approximations, often in terms of the $\ell_2$ error. Due to space limits, we shall focus on the ones that are closest to ours, i.e. the ones that deliver $\ell_{\infty}$ guarantees in the tabular setting.


\paragraph{Finite-sample $\ell_\infty$-based guarantees for synchronous Q-learning and TD learning.} The sample complexities derived in prior literature often rely crucially on the choices of learning rates. \citet{even2003learning} studied the sample complexity of Q-learning with linear learning rates $1/t$ or polynomial learning rates $1/t^{\omega}$, which scales as $\widetilde{O}(\frac{|\cS||\cA|}{(1-\gamma)^5 \varepsilon^{2.5}})$ when optimized w.r.t.~the effective horizon (attained when $\omega =4/5$). The resulting sample complexity, however, is sub-optimal in terms of its dependency on not only  $\frac{1}{1-\gamma}$ but also the target accuracy level $\varepsilon$. \citet{beck2012error} investigated the case of constant learning rates; however, their result suffered from an additional factor of $|\cS||\cA|$, which could be prohibitively large in practice. More recently, \citet{wainwright2019stochastic,chen2020finite} further analyzed the sample complexity of Q-learning with either constant learning rates or linearly rescaled learning rates, leading to the state-of-the-art bound $\widetilde{O}\big(\frac{|\cS||\cA|}{(1-\gamma)^5\varepsilon^2}\big)$. However, this result remains sub-optimal in terms of its scaling with $\frac{1}{1-\gamma}$.  See Table~\ref{table:comparisons} for details. 
In the special case with $|\cA|=1$, 
the recent works \cite{khamaru2020temporal,mou2020linear} 
developed instance-dependent results for TD learning with Polyak-Ruppert averaging, and studied the local (sub)-optimality of TD learning in a different local minimax framework.   
% Also worth noting is the works of \citet{lakshminarayanan2018linear,bhandari2021finite},  which studied the performance of (averaged) TD learning in the presence of independence samples with properly weighted marginal state distributions; the results therein, however, were $\ell_2$-based estimation guarantees. 


\paragraph{Finite-sample $\ell_\infty$-based guarantees for asynchronous Q-learning and TD learning.}
Moving beyond the synchronous model, \cite{even2003learning,beck2012error,qu2020finite,li2020sample, shah2018q, chen2021lyapunov}
developed non-asymptotic convergence guarantees for the asynchronous setting, where the data samples take the form of a single Markovian trajectory (following some behavior policy) and only a single state-action pair is updated in each iteration. A similar scaling of $\widetilde{O}\big(\frac{1}{(1-\gamma)^5}\big)$  also showed up in the state-of-the-art sample complexity bounds for asynchronous Q-learning \citep{li2020sample}, and our theory is the first to sharpen it to $\widetilde{O}\big(\frac{1}{(1-\gamma)^4}\big)$. 
%The analysis framework developed in this paper might have implications on how to sharpen the dependency of sample complexity on $\frac{1}{1-\gamma}$ for asynchronous Q-learning. 
When it comes to the special case with $|\cA|=1$,  the non-asymptotic performance guarantees TD learning with Markovian sample trajectories (assuming that the behavior policy coincides with the target policy) have been recently derived by \cite{bhandari2021finite,srikant2019finite,mou2020linear}. 



% As can be seen, the dependencies with the effective horizon $H$ seem to be suffer from a barrier of $H^5$ for the vanilla Q-learning algorithms in all existing bounds, which the current paper aims to break via introducing a new analysis strategy. 



\paragraph{Finite-sample $\ell_\infty$-based guarantees of other Q-learning variants.}  
With the aim of alleviating the sub-optimal dependency on the effective horizon in vanilla Q-learning and improving sample efficiency, several variants of Q-learning have been proposed and analyzed. \citet{azar2011reinforcement} proposed speedy Q-learning, which achieves a sample complexity of $\widetilde{O} \big( \frac{|\cS||\cA|}{ (1-\gamma)^4\varepsilon^2 } \big)$ at the expense of doubling the computation and storage complexity. Our result on vanilla Q-learning matches that of speedy Q-learning in an order-wise sense. In addition, \citet{wainwright2019variance} proposed a variance-reduced Q-learning algorithm that is shown to be minimax optimal in the range $\epsilon \in (0,1)$ with a sample complexity $\widetilde{O} \big( \frac{|\cS||\cA|}{ (1-\gamma)^3\varepsilon^2 } \big)$, which was subsequently generalized to the asynchronous setting by \citet{li2020sample}. The $\ell_\infty$ statistical bounds for variance-reduced TD learning have been investigated in \citet{khamaru2020temporal} for the synchronous setting, and in \citet{li2020sample} for the asynchronous setting. Last but not least, \citet{xiong2020finitetime} established the finite-sample convergence of double Q-learning following the framework of \cite{even2003learning}; however, it is unclear whether  double Q-learning can provably outperform vanilla Q-learning in terms of the sample efficiency.

% 


%\paragraph{Finite-sample $\ell_\infty$ guarantees of model-based algorithms.} Model-based algorithms, which perform policy evaluation and planning based on the empirical MDP constructed from the samples, constitute another 
%appealing approach in RL.  It is shown that the plug-in approach achieves the minimax-optimal sample complexity in the synchronous setting for learning the optimal Q-function in \cite{azar2013minimax, pananjady2019value}, and the optimal policy in \cite{agarwal2019optimality,li2020breaking}. \citet{sidford2018near} proposed a randomized variant of Q-value iteration together with variance reduction and certain monotonicity preserving to learn the optimal policy in a minimax-optimal manner.


\paragraph{Others.} 
 There are also several other strands of related papers that tackle model-free algorithms but do not pursue $\ell_{\infty}$-based non-asymptotic guarantees. For instance, \citet{bhandari2018finite,lakshminarayanan2018linear,srikant2019finite,gupta2019finite,doan2019finite,wu2020finite,xu2019reanalysis,chen2019performance,xu2019two} developed finite-sample (weighted) $\ell_2$ convergence guarantees for several model-free algorithms, which also allow one to accommodate linear function approximation as well as off-policy evaluation.  Another line of recent work \citep{jin2018q,li2021breaking,bai2019provably,zhang2020almost} considered the sample efficiency of Q-learning type algorithms paired with proper exploration strategies (e.g., upper confidence bounds) under the framework of regret analysis.   
The asymptotic behaviors of some variants of Q-learning, e.g., double Q-learning \citep{weng2020mean} and relative Q-learning \citep{devraj2020q} are also studied. 
The effect of more general function approximation schemes (e.g., certain families of neural network approximations) has been studied in \citet{yang2019theoretical,murphy2005generalization,cai2019neural,wai2019variance,xu2020finite} as well. These are beyond the scope of the present paper. 


 


%  Another contrasting approach for learning the optimal Q-function is the class of model-based algorithms, which has been shown to enjoy minimax-optimal sample complexity in the synchronous setting. More precisely, it is known that by planning over an empirical MDP constructed from $\widetilde{O}\big(\frac{|\cS||\cA|}{(1-\gamma)^3\varepsilon^2}\big)$ samples, we are guaranteed to find not only an $\varepsilon$-optimal Q-function but also an $\varepsilon$-optimal policy \citep{azar2013minimax,agarwal2019optimality,li2020breaking}. It is worth emphasizing that the minimax optimality of model-based approach has been shown to  hold for the entire $\varepsilon$-range; in comparison, the sample optimality of the model-free approach has only been shown for a smaller range of accuracy level $\varepsilon$ in the synchronous setting.  



lowerbound.tex
\section{Lower bound: sub-optimality of Q-learning (Theorem~\ref{thm:LB-example})}
\label{sec:lower-bounds}



In this section, we establish the lower bound claimed in Theorem~\ref{thm:LB-example} by analyzing Q-learning for the MDP instance constructed in Section~\ref{sec:MDP-construction-hard}. Without loss of generality, we assume
%
\begin{equation} \label{eq:logT_ub}
\log T\leq \frac{1}{1-\gamma}
\end{equation} 
%
throughout the proof; otherwise the lower bound in Theorem~\ref{thm:LB-example} is worse than the minimax lower bound $\frac{1}{(1-\gamma)^3T}$ in \citet{azar2013minimax}. 



Throughout, we shall use $P_t$ to represent the sample transitions such that for any triple $(s,a,s')$,
%
\begin{align}
	P_t(s'\mymid s,a) \coloneqq \begin{cases}  1, \quad & \text{if } s_t(s,a)=s', \\ 0, & \text{otherwise}, \end{cases}
\end{align}
%
where $s_t(s,a)$ stands for the sample collected in the $t$-th iteration (see \eqref{defn:empirical-Bellman-t-inf}). 
Recognizing that state  2 is associated with a singleton action space, we shall often write
%
\[
	P_t(s'\mymid 2) \coloneqq   P_t(s'\mymid 2,1)
\]
%
for notational simplicity. 



\subsection{Key quantities related to learning rates}

We find it convenient to define the following quantities (by abuse of notation) 
%
\begin{subequations}
\label{eq:defn-eta-kt-hard}
\begin{align}
\eta_k^{(t)} &\defn \eta_k\prod_{i = k+1}^t \big(1-\eta_i(1-\gamma p)\big) \qquad\text{for any }1 \le k < t, \\
\eta_0^{(t)} &\defn \prod_{i = 1}^t \big(1-\eta_i(1-\gamma p)\big), \\
\eta_t^{(t)} &\defn \eta_t. 
\end{align}
\end{subequations}
%
It is helpful to establish several basic properties about these quantities. 
As can be easily verified, 
%
\begin{align}
	\eta_0^{(t)} + (1-\gamma p)\sum_{k = 1}^t \eta_k^{(t)} = \prod_{i=1}^{t}(1-\widehat{\eta}_{i})+\widehat{\eta}_{1}\prod_{i=2}^{t}(1-\widehat{\eta}_{i})+\widehat{\eta}_{2}\prod_{i=3}^{t}(1-\widehat{\eta}_{i})+\cdots+\widehat{\eta}_{t-1}(1-\widehat{\eta}_{t})+\widehat{\eta}_{t}=1,
	\label{eq:eta-sum-hat}
\end{align}
%
where we denote $\widehat{\eta}_{i} \defn \eta_{i}(1-\gamma p)$ to simplify notation. 
Similarly, for any given integer $0\leq \tau < t$ one has
%
\begin{align}
	\label{eq:eta-sum-hat-tau}
	\prod_{i = \tau+1}^t \big(1-\eta_i(1-\gamma p)\big) + (1-\gamma p)\sum_{k = \tau+1}^t \eta_k^{(t)} = 1.
\end{align}
%


\subsection{Preliminary calculations}

Before moving forward, we record several basic relations as a result of the Q-learning update rule.  


\subsubsection{Basic update rules and expansion}
Given that $Q_0 = V_0 = 0$ and that state 0 is absorbing, the  update rule \eqref{eqn:q-learning} gives
%
\begin{align} \label{eq:vt-0}
V_t(0) = Q_t(0,1) = \big(1-\eta_t(1-\gamma)\big)Q_{t-1}(0,1) = \prod_{i = 1}^t \big(1-\eta_i(1-\gamma)\big)Q_0(0,1) = 0
\end{align}
%
for all $t\geq 1$. 
Regarding state 2, the update rule \eqref{eqn:q-learning} taken together with \eqref{eq:vt-0} leads to
%
\begin{align} 
	V_t(2) = Q_t(2,1)  & = \big(1-\eta_t\big)Q_{t-1}(2,1) + \eta_t\big\{ r(2,1)+ \gamma P_t(2 \mymid 2)V_{t-1}(2) + \gamma P_t(0 \mymid 2)V_{t-1}(0) \big\} \notag\\
	& = \big(1-\eta_t\big)V_{t-1}(2) + \eta_t\big\{ 1 + \gamma P_t(2 \mymid 2)V_{t-1}(2)\big\},  \label{eq:vt-2}
\end{align}
and for state 3,
\begin{align} 
	V_t(3) = Q_t(3,1)  & = \big(1-\eta_t\big)Q_{t-1}(3,1) + \eta_t\big\{ r(3,1)+ \gamma V_{t-1}(3) \big\} \notag\\
	& = \big(1-\eta_t(1-\gamma)\big)V_{t-1}(3) + \eta_t .  \label{eq:vt-3}
\end{align}
%
Similarly, one also has
%
\begin{subequations}
\label{eq:Qt-1}
\begin{align}
	Q_t(1, 1) &= (1-\eta_t)Q_{t-1}(1, 1) + \eta_t\big\{ 1 + \gamma P_{t}( 1 \mymid 1, 1) V_{t-1}(1)\big\}, \\
	Q_t(1, 2) &= (1-\eta_t)Q_{t-1}(1, 2) + \eta_t\big\{ 1 + \gamma P_{t}( 1 \mymid 1, 2) V_{t-1}(1)\big\}.
\end{align}
\end{subequations}
In what follows, we shall first determine a crude range for certain quantities relates to the learning rates $\eta_t$, and then combine this with the above relations to establish the desired result.


Next, we record some elementary decomposition of $V_t(2)$.  For any iteration $t$ and $\tau<t$, one can continue the derivation in \eqref{eq:vt-2} to obtain
%
\begin{align}
	V_t(2) &= \big(1-\eta_t(1-\gamma p)\big)V_{t-1}(2) + \eta_t\big\{ 1+ \gamma \big( P_t(2 \mymid 2) - p \big) V_{t-1}(2)\big\} \nonumber\\
	&= \prod_{i = \tau+1}^t \big(1-\eta_i(1-\gamma p)\big)V_{\tau}(2) + \sum_{k = \tau+1}^t\eta_k\prod_{i = k+1}^t \big(1-\eta_i(1-\gamma p)\big)\big\{ 1+ \gamma \big(P_k(2 \mymid 2) - p \big)V_{k-1}(2)\big\} \nonumber\\
	&=  \prod_{i = \tau+1}^t \big(1-\eta_i(1-\gamma p)\big)V_{\tau}(2) + \sum_{k = \tau+1}^t \eta_k^{(t)}  + \sum_{k = \tau+1}^t \eta_k^{(t)}  \gamma \big(P_k(2 \mymid 2) - p \big)V_{k-1}(2) \nonumber\\
	&=  \prod_{i = \tau+1}^t \big(1-\eta_i(1-\gamma p)\big)V_{\tau}(2) + \frac{1 - \prod_{i = \tau+1}^t \big(1-\eta_i(1-\gamma p)\big) }{1-\gamma p}  + \sum_{k = \tau+1}^t \eta_k^{(t)}  \gamma \big(P_k(2 \mymid 2) - p \big)V_{k-1}(2) \nonumber\\
	&= \frac{1}{1-\gamma p} - \prod_{i = \tau+1}^t \big(1-\eta_i(1-\gamma p)\big)\left[\frac{1}{1-\gamma p} - V_{\tau}(2)\right] + \sum_{k = \tau+1}^t\eta_k^{(t)}\gamma \big(P_k(2\mymid 2) - p\big) V_{k-1}(2), 
	\label{eq:expansion-Vt2-tau}
\end{align}
%
where the penultimate line arises from \eqref{eq:eta-sum-hat-tau}.
In particular, in the special case where $\tau=0$ (so that $V_{\tau}(2)=V_{0}(2)=0$), this simplifies to
%
\begin{align} 
	V_t(2) = \frac{1-\eta_0^{(t)}}{1-\gamma p} + \sum_{k = 1}^t\eta_k^{(t)}\gamma \big(P_k(2 \mymid 2) - p \big)V_{k-1}(2),  
	\label{eq:Vt2-expansion-case1}
\end{align}
%
which relies on the definition of $\eta_0^{(t)}$ in \eqref{eq:defn-eta-kt-hard}. 
With similar derivation, \eqref{eq:vt-3} leads to
\begin{align} 
	V_t(3) = \frac{1}{1-\gamma}\Big[1-\prod_{i = 1}^T \big(1-\eta_i(1-\gamma)\big)\Big] = V^{\star}(3) - \frac{1}{1-\gamma}\prod_{i = 1}^T \big(1-\eta_i(1-\gamma)\big).
	\label{eq:Vt3-expansion-case3}
\end{align}


\subsubsection{Mean and variance of $V^{\star}(2) - V_T(2)$ }

We start by computing the mean $V^{\star}(2) - \mathbb{E}[V_t(2)]$. 
From the construction \eqref{eq:construction-hard-MDP}, it is easily seen that $\mathbb{E}[ P_k(2 \mymid 2) ] = p$, which together with the identity \eqref{eq:Vt2-expansion-case1} leads to
%
\begin{align}
	\label{eq:EV2}
	\mathbb{E} \big[V_T(2)\big] = \frac{1-\eta_0^{(T)}}{1-\gamma p}
	\qquad \text{and}\qquad
	V^{\star}(2) - \mathbb{E} \big[V_T(2)\big] = \frac{\eta_0^{(T)}}{1-\gamma p}. 
\end{align}
%
Similarly, applying the above argument to \eqref{eq:expansion-Vt2-tau} and rearranging terms, we immediately arrive at
%
%
\begin{align}
	 V^{\star}(2) - \mathbb{E}\big[V_{T}(2)\big] 
	= \prod_{i = \tau+1}^T \big(1-\eta_i(1-\gamma p)\big)\left[\frac{1}{1-\gamma p} - \mathbb{E}\big[ V_{\tau}(2)\big]\right]
	\label{eq:Vstar2-V2-expansion-Case2}
\end{align}
%
for any integer $0\leq \tau <T$. 


Next, we develop a lower bound on the variance $\mathsf{Var}\big(V_T(2)\big)$. 
Towards this end, consider first a martingale sequence $\{Z_k\}_{0\leq k\leq T}$ adapted to 
a filtration $\mathcal{F}_0\subseteq  \mathcal{F}_1 \subseteq \cdots \subseteq \mathcal{F}_T$, 
namely,  $\mathbb{E} [Z_{k+1} \mymid \mathcal{F}_{k}]=0$ and $\mathbb{E} [Z_k \mymid \mathcal{F}_{k}]=Z_k$ for all $0\leq k\leq T$. 
In addition, consider any $0\leq \tau < T$, and let $W_{0}$ be a random variable such that $\mathbb{E}[W_{0}\mymid \mathcal{F}_{\tau}]=W_0$. 
Then the law of total variance together with basic martingale properties tells us that 
%
\begin{align}
\mathsf{Var}\left(W_{0}+\sum_{k=\tau+1}^{T}Z_{k}\right) & =\mathbb{E}\left[\mathsf{Var}\left(W_{0}+\sum_{k=\tau+1}^{T}Z_{k}\mid\mathcal{F}_{T-1}\right)\right]+\mathsf{Var}\left(\mathbb{E}\left[W_{0}+\sum_{k=\tau+1}^{T}Z_{k}\mid\mathcal{F}_{T-1}\right]\right) \notag\\
 & =\mathbb{E}\big[\,\mathsf{Var}\left(Z_{T}\mid\mathcal{F}_{T-1}\right)\big]+\mathsf{Var}\left(W_{0}+\sum_{k=\tau+1}^{T-1}Z_{k}\right)=\cdots \notag\\
 & =\sum_{k=\tau+1}^{T}\mathbb{E}\big[\,\mathsf{Var}\left(Z_{k}\mid\mathcal{F}_{k-1}\right)\big]+\mathsf{Var}\left(W_{0}\right)\geq\sum_{k=\tau+1}^{T}\mathbb{E}\big[\,\mathsf{Var}\left(Z_{k}\mid\mathcal{F}_{k-1}\right)\big].
	\label{eq:variance-expansion-martingale}
\end{align}
%
Consequently, for any $0\leq \tau < T-1$, it follows from the decomposition \eqref{eq:expansion-Vt2-tau} (with $\tau$ replaced by $\tau+1$) that
%
\begin{align}
\mathsf{Var}\big(V_{T}(2)\big) & \ge\mathbb{E}\left[\sum_{k=\tau+2}^{T}\mathsf{Var}\Big(\eta_{k}^{(T)}\gamma\big(P_{k}(2\mymid2)-p\big)V_{k-1}(2)\mymid V_{k-1}(2)\Big)\right] \notag\\
 & =\sum_{k=\tau+2}^{T}\big(\eta_{k}^{(T)}\gamma\big)^{2}p(1-p)\mathbb{E}\big[\big(V_{k-1}(2)\big)^{2}\big] \notag\\
 & \geq\frac{\left(1-\gamma\right)\left(4\gamma-1\right)}{9}\cdot\frac{1}{4(1-\gamma)^{2}}\sum_{k=\tau+2}^{T}\big(\eta_{k}^{(T)}\big)^{2} \notag\\
 & =\frac{ 4\gamma-1}{36\left(1-\gamma\right)}\sum_{k=\tau+2}^{T}\big(\eta_{k}^{(T)}\big)^{2},
	\label{eq:Var-VT-2-LB-123}
\end{align}
%
where the first identity relies on the fact that $P_k(2\mymid 2)$ is a Bernoulli random variable with mean $p$, and the inequality comes from the definition of $\tau$ (see \eqref{eq:defn-tau-Case2-LB}) and the choice of $p$ (see \eqref{defn-parameter-p}). As an implication, the sum of squares of $\eta_{k}^{(T)}$ plays a crucial role in determining the variance of $V_{T}(2)$.



\subsection{Lower bounds for three cases}


\subsubsection{Case 1: small learning rates ($\ceta \geq \log T$ or $0\leq \eta \leq \frac{1}{(1-\gamma)T}$)}

 




In this case, we focus on lower bounding $V^{\star}(2) - \mathbb{E}\big[ V_T(2)\big]$.  
In view of this identity \eqref{eq:EV2}, this boils down to controlling $\eta_0^{(T)}$. 


Suppose that $\ceta > \log T$ (for rescaled linear learning rates) or $0\leq \eta < \frac{1}{(1-\gamma)T}$ (for constant learning rates).  
A little algebra then gives 
%
\begin{align}
\label{eq:etat-gammap-UB1}
\eta_{t}(1-\gamma p) & \leq\begin{cases}
\frac{1-\gamma p}{(1-\gamma)t\log T}=\frac{4}{3t\log T}\leq\frac{1}{2},\qquad & \text{if }\eta_{t}=\frac{1}{1+\ceta(1-\gamma)t}\\
\frac{1-\gamma p}{(1-\gamma)T}=\frac{4}{3T}\leq\frac{1}{2}, & \text{if }\eta_{t}=\eta
\end{cases}
\end{align}
%
for any $t\geq 1$, provided that $T\geq 15$. Consequently, one can derive
%
\begin{align}
	\log \eta_0^{(T)} 
	%= \log \Bigg[\prod_{i = 1}^T \big(1-\eta_i(1-\gamma p)\big)\Bigg] 
	= \sum_{i = 1}^T \log \big(1-\eta_i(1-\gamma p)\big) \ge - 1.5\sum_{i = 1}^T \eta_i(1-\gamma p) \ge -2,
	\label{eq:log-eta-0T-LB}
\end{align}
%
where the first inequality holds due to the elementary fact $\log(1-x)\geq -1.5x$ for all $0\leq x \leq 0.5$, 
and the last inequality follows from the following bound (which makes use of \eqref{eq:etat-gammap-UB1})
%
\[
\sum_{i=1}^{T}\eta_{i}(1-\gamma p)\leq\begin{cases}
\frac{3}{4\log T}\sum_{i=1}^{T}\frac{1}{i}\leq1, & \text{if }\eta_{t}=\frac{1}{1+\ceta(1-\gamma)t}\\
\frac{4}{3T}\sum_{i=1}^{T}1=\frac{4}{3}, & \text{if }\eta_{t}=\eta.
\end{cases}
\]
%
Combining the above result with the properties \eqref{eq:EV2} and \eqref{eq:log-eta-0T-LB} then yields
%
\begin{equation}
\label{eqn:Vstar-lb-violin}
	%\mathbb{E} \big[ \, \big| V^{\star}(2) - V_T(2) \big| \, \big]  \geq 
	V^{\star}(2) - \mathbb{E} \big[V_T(2)\big]  =  \frac{\eta_0^{(T)}}{1-\gamma p} \ge  \frac{e^{-2}}{1-\gamma p} = \frac{3}{4e^2(1-\gamma)}.
\end{equation}
%
This taken together with \eqref{eq:bias-variance-decomposition} gives
%
\begin{equation}
	\label{eq:MSE-V2-case1}
	\mathbb{E}\big[ \big( V^{\star}(2) - V_T(2) \big)^2 \big] 
	\geq \big( V^{\star}(2) - \mathbb{E} \big[V_T(2)\big] \big)^2 
	\geq \frac{9}{16e^4(1-\gamma)^2}.
\end{equation}
%



\subsubsection{Case 2: large learning rates ($\ceta \leq 1-\gamma$ or $ \eta \geq \frac{1}{(1-\gamma)^2T}$)}

By virtue of \eqref{eq:Vstar2-V2-expansion-Case2},  the mean gap $V^{\star}(2) - \mathbb{E}\big[V_{T}(2)\big]$ depends on two factors: (i) the choice of the learning rates, and (ii) the gap between $\frac{1}{1-\gamma p}$ and $\mathbb{E}\big[ V_{\tau}(2)\big]$, where $\tau$ is an integer obeying $0\leq \tau <T$. 
To control the factor (ii), we need to choose $\tau$ properly. 
Let us start by considering the simple scenario with $\mathbb{E}\big[ \big( V_{T}(2) \big)^2 \big] < \frac{1}{4(1-\gamma)^2}$, for which we have
%
\begin{align}
	%\mathbb{E} \big[ \, \big| V^{\star}(2) - V_T(2) \big| \, \big]  \geq  
	V^{\star}(2) -  \mathbb{E}\big[  V_{T}(2)  \big] 
	\geq \frac{3}{4(1-\gamma)} -  \sqrt{\mathbb{E}\big[ \big( V_{T}(2) \big)^2 \big]} 
	\geq \frac{1}{4(1-\gamma)}. 
	\label{eq:mean-diff-large-tau-T}
\end{align}
%
Here, we have used \eqref{eq:optimal-V-Q-hardMDP} and the elementary fact $\mathbb{E}[X]\leq \sqrt{\mathbb{E}[X^2]}$. Consequently, it remains to look at the scenario obeying $\mathbb{E}\big[ \big( V_{T}(2) \big)^2 \big] \geq \frac{1}{4(1-\gamma)^2}$, towards which we propose to set $\tau$ as follows
%
\begin{align}
	\label{eq:defn-tau-Case2-LB}
	\tau \defn \min\left\{0\leq \tau' \leq T-1 \,\,\Big|\,\, \mathbb{E}\big[ \big( V_{t}(2) \big)^2 \big] 
	\ge \frac{1}{4(1-\gamma)^2} ~\text{for all } \tau'+1 \leq t \leq  T \right\} .
\end{align}
%
Clearly, $\tau$ is well-defined in this scenario and obeys (in view of both \eqref{eq:defn-tau-Case2-LB} and the initialization $V_0=0$)
%
\begin{align}
	\label{eq:E-Vtau-UB-case2}
	\mathbb{E}\big[ \big( V_{\tau}(2) \big)^2 \big] 
	< \frac{1}{4(1-\gamma)^2} .
\end{align}
%
Our analysis for this scenario is divided into three subcases based on the size of the learning rates. 





\paragraph{\em Case 2.1.} Consider the case where
%
\begin{align}
	\label{eq:case-21-condition}
\prod_{i = \tau+1}^T \big(1-\eta_i(1-\gamma p)\big) \geq \frac{1}{2}.
\end{align}
%
Invoke \eqref{eq:Vstar2-V2-expansion-Case2} to deduce that
%
\begin{align*}
	%\mathbb{E} \big[ \, \big| V^{\star}(2) - V_T(2) \big| \, \big]  &\geq  
	V^{\star}(2) - \mathbb{E}\big[V_{T}(2)\big] &= \prod_{i = \tau+1}^T \big(1-\eta_i(1-\gamma p)\big)\Big[\frac{1}{1-\gamma p} - \mathbb{E}\big[V_{\tau}(2)\big]\Big] \\
	&\ge \prod_{i = \tau+1}^T \big(1-\eta_i(1-\gamma p)\big)\Big[\frac{3}{4(1-\gamma) } - \sqrt{\mathbb{E}\big[ \big( V_{\tau}(2) \big)^2 \big] }\Big] \\
&\ge \prod_{i = \tau+1}^T \big(1-\eta_i(1-\gamma p)\big)\frac{1}{4(1-\gamma)} \ge \frac{1}{8(1-\gamma)}, 
\end{align*}
%
where the second line makes use of the definition \eqref{defn-parameter-p} and the elementary fact $\mathbb{E}[X]\leq \sqrt{\mathbb{E}[X^2]}$, 
and the last line relies on the inequalities \eqref{eq:E-Vtau-UB-case2} and \eqref{eq:case-21-condition}. 



\paragraph{\em Case 2.2.} We now move on to the case where 
%
\begin{align}
0 \le \prod_{i = \tau+1}^T \big(1-\eta_i(1-\gamma p)\big) \le \frac{1}{2}.
	\label{eq:prod-etai-UB-lower-bound}
\end{align}
%
We intend to demonstrate that the variance of $V_T(2)$ --- and hence the typical size of its fluctuation --- is too large. 
In view of the observation \eqref{eq:Var-VT-2-LB-123}, it boils down to lower bounding $\sum_{k=\tau+2}^{T}\big(\eta_{k}^{(T)}\big)^{2}$, which we accomplish as follows. 
%
\begin{itemize}
	\item Consider constant learning rates $\eta_k=\eta$, and suppose that $\eta$ obeys $ \frac{1}{(1-\gamma)^2T} < \eta \leq 1< \frac{1}{1-\gamma p}$. 
		It is readily seen that $\eta_{k}^{(T)}=\eta\big(1-\eta(1-\gamma p)\big)^{T-k}$ for any $k\geq 1$. 
We claim that it suffices to focus on the scenario where
%
\begin{align}
	\tau \leq T-2. 
	\label{eq:assumption-tau-T-minus-2}
\end{align}
%
In fact, if $\tau \geq T-1$, then the definition \eqref{eq:defn-tau-Case2-LB} of $\tau$ necessarily requires that
%
\[
	\mathbb{E}\big[V_{T-1}(2)\big] \leq \sqrt{ \mathbb{E}\big[ \big( V_{T-1}(2) \big)^2 \big] } < \frac{1}{2(1-\gamma)}.
\]
%
In view of \eqref{eq:EV2} (with $T$ replaced by $T-1$), a little algebra shows that this is equivalent to $\big(1-\eta(1-\gamma p)\big)^{T-1} \ge 1/3$,
and hence $\big(1-\eta(1-\gamma p)\big)^{T} \ge 1/9$.
In turn, this combined with \eqref{eq:EV2} leads to
	%
	\begin{align}
		%\mathbb{E} \big[ \, \big| V^{\star}(2) - V_T(2) \big| \, \big] \geq 
		V^{\star}(2) - \mathbb{E}\big[V_{T}(2)\big] 
		= \frac{ \big(1-\eta(1-\gamma p)\big)^{T}}{1-\gamma p} 
		= \frac{3 \big(1-\eta(1-\gamma p)\big)^{T}}{4( 1-\gamma )} 
		\ge \frac{1}{12(1-\gamma)},
		\label{eq:large-tau-case-mean-diff}
	\end{align}
	%
which already suffices for our purpose. 

Next, assuming that \eqref{eq:assumption-tau-T-minus-2} holds, one can derive 
	%
\begin{align}
\sum_{k=\tau+2}^{T}\big(\eta_{k}^{(T)}\big)^{2} & =\sum_{k=\tau+2}^{T}\eta^{2}\big(1-\eta(1-\gamma p)\big)^{2(T-k)}=\frac{\eta^{2}\big[1-\big(1-\eta(1-\gamma p)\big)^{2(T-\tau-1)}\big]}{1-\big(1-\eta(1-\gamma p)\big)^{2}} \notag\\
	& \ge\frac{\eta^{2}/2}{1-\big( 1- \eta(1-\gamma p)\big)^{2}}\ge\frac{3\eta}{16(1-\gamma)},
	\label{eq:eta-k-prod-case-1234}
\end{align}
%
where the first inequality holds since (from the assumptions \eqref{eq:prod-etai-UB-lower-bound} and $\tau\leq T-2$)
\[
	0\leq \big(1-\eta(1-\gamma p)\big)^{2(T-\tau-1)} \le \big(1-\eta(1-\gamma p)\big)^{T-\tau} 
		= \prod_{i = \tau+1}^T \big(1-\eta_i(1-\gamma p)\big) \le \frac{1}{2}, 
\]
%
and the last inequality follows since 
%
\[
	0 \le 1-\big( 1- \eta(1-\gamma p)\big)^{2} =  1-\Big(1-\frac{4\eta(1-\gamma)}{3}\Big)^{2} \le \frac{8\eta(1-\gamma)}{3}.
\]
%
Substituting \eqref{eq:eta-k-prod-case-1234} into \eqref{eq:Var-VT-2-LB-123}, we obtain
%
\begin{align}
\mathsf{Var}\big(V_{T}(2)\big) & \ge\frac{4\gamma-1}{36(1-\gamma)}\sum_{k=\tau+1}^{T}\big(\eta_{k}^{(T)}\big)^{2}\ge\frac{2}{36(1-\gamma)}\cdot\frac{3\eta}{16(1-\gamma)} \notag\\
 & =\frac{\eta}{96(1-\gamma)^{2}}\geq\frac{1}{96(1-\gamma)^{4}T} ,\label{eq:case2_constant_stepsize}
\end{align}
 provided that $\gamma \geq 3/4$ (so that $4\gamma - 1 \geq 2$). 
Here, the last inequality is valid since either $\eta\geq \frac{1}{(1-\gamma)^2T}$.


\item We then move on to linearly rescaled learning rates with $\eta_t = \frac{1}{1+\ceta(1-\gamma)t}$ for some $0\leq \ceta<1-\gamma$. 
Towards this, we first make the observation that 
%
\begin{align}
\frac{\eta_{k-1}^{(T)}}{\eta_{k}^{(T)}} & =\frac{\eta_{k-1}\big(1-\eta_{k}(1-\gamma p)\big)}{\eta_{k}}=\frac{1-\frac{4}{3}(1-\gamma)\eta_{k}}{1-\big(\frac{1}{\eta_{k}}-\frac{1}{\eta_{k-1}}\big)\eta_{k}}=\frac{1-\frac{4}{3}(1-\gamma)\eta_{k}}{1-\ceta(1-\gamma)\eta_{k}}=1-\frac{\big(\frac{4}{3}-\ceta\big)(1-\gamma)\eta_{k}}{1-\ceta(1-\gamma)\eta_{k}} \notag\\
 & \le1-(1-\gamma)\eta_{k}\le1-(1-\gamma)\eta_{T},
 \label{eq:ratio-etakT-eta-kminus1}
\end{align}	
%
with the proviso that $\ceta < 1-\gamma \leq 1/3$ (as long as $\gamma \geq 2/3$).  
By defining $\tau' \defn T-\frac{1}{(1-\gamma)\eta_{T}}$, one can deduce that
%
\begin{align}
\sum_{k=\tau+2}^{T}\big(\eta_{k}^{(T)}\big)^{2} & \ge\sum_{k=\max\{\tau+2,\tau'+1\}}^{T}\big(\eta_{k}^{(T)}\big)^{2}\ge\frac{1}{T-\max\{\tau+1,\tau'\}}\left[\sum_{k=\max\{\tau+2,\tau'+1\}}^{T}\eta_{k}^{(T)}\right]^{2} \notag\\
 & \ge(1-\gamma)\eta_{T}\left[\sum_{k=\max\{\tau+2,\tau'+1\}}^{T}\eta_{k}^{(T)}\right]^{2} ,
	\label{eq:sum-eta-kT-lower-bound-234}
	%\ge\frac{\eta_{T}}{211(1-\gamma)},
\end{align}
%
where the penultimate inequality comes from the Cauchy-Schwarz inequality. 
In addition, recognizing that $\eta_{k_1}^{(T)} \leq \big(1-(1-\gamma)\eta_{T}\big)^{k_2-k_1}\eta_{k_2}^{(T)}$ for any $k_2 \ge k_1$ (see \eqref{eq:ratio-etakT-eta-kminus1}), one has 
%
\begin{align*}
	\sum_{k=\tau'+1}^{T}\eta_{k}^{(T)} & = \sum_{k=\tau'+1}^{T}\eta_{k}^{(T)}, \notag\\
\sum_{k=\max\{2\tau'-T+1,1\}}^{\tau'}\eta_{k}^{(T)} & \leq\big(1-(1-\gamma)\eta_{T}\big)^{T-\tau'}\sum_{k=\tau'+1}^{T}\eta_{k}^{(T)}, \notag\\
\sum_{k=\max\{3\tau'-2T+1,1\}}^{2\tau'-T}\eta_{k}^{(T)} & \leq \big(1-(1-\gamma)\eta_{T}\big)^{2(T-\tau')}\sum_{k=\tau'+1}^{T}\eta_{k}^{(T)} , \notag\\
 & \cdots
\end{align*}
%
Summing these inequalities up and rearranging terms, we reach
%
\begin{align*}
\sum_{k=\tau'+1}^{T}\eta_{k}^{(T)} & \ge\frac{\sum_{k=1}^{T}\eta_{k}^{(T)}}{1+\big(1-(1-\gamma)\eta_{T}\big)^{T-\tau'}+\big(1-(1-\gamma)\eta_{T}\big)^{2(T-\tau')}+\ldots}\geq\frac{\sum_{k=1}^{T}\eta_{k}^{(T)}}{\frac{1}{1-\big(1-(1-\gamma)\eta_{T}\big)^{T-\tau'}}}\\
 & =\left(1-\big(1-(1-\gamma)\eta_{T}\big)^{T-\tau'}\right)\sum_{k=1}^{T}\eta_{k}^{(T)}\geq (1-e^{-1})\sum_{k=1}^{T}\eta_{k}^{(T)},
\end{align*}
%
which relies on the fact $\big(1-(1-\gamma)\eta_{T}\big)^{T-\tau'} = \big(1-1/(T-\tau')\big)^{T-\tau'} \le e^{-1}$ (using the definition of $\tau'$).
%
Consequently, it is easily seen that
%
\begin{align*}
\sum_{k=\max\{\tau+2,\tau'+1\}}^{T}\eta_{k}^{(T)} 
	& = \min \left\{ \sum_{k=\tau+2}^{T}\eta_{k}^{(T)} , \sum_{k=\tau'+1}^{T}\eta_{k}^{(T)} \right\} 
	\ge (1-e^{-1})\sum_{k = \tau+2}^T \eta_k^{(T)} \\
	& \overset{\text{(i)}}{=} (1-e^{-1})\left[1-\prod_{i = \tau+2}^t \big(1-\eta_i(1-\gamma p)\big)\right]\frac{1}{1-\gamma p} \\
	&\overset{\text{(ii)}}{\ge} \left[1-\frac{1}{2\big(1-\eta_{\tau+1}(1-\gamma p)\big)}\right]\frac{1-e^{-1}}{1-\gamma p} 
	\overset{\text{(iii)}}{\ge} \frac{1-e^{-1}}{4(1-\gamma p)} \ge \frac{3}{32(1-\gamma )} .
\end{align*}
%
Here, (i) and (ii) follow from \eqref{eq:eta-sum-hat-tau} and \eqref{eq:prod-etai-UB-lower-bound}, respectively, while (iii) holds since 
$$\eta_{\tau+1}(1-\gamma p) \le  1-\gamma p = \frac{4(1-\gamma)}{3} \le \frac{1}{3}$$ as long as $\gamma \ge 3/4$. 
%
Substitution into \eqref{eq:sum-eta-kT-lower-bound-234} yields
%
\begin{align}
\sum_{k=\tau+2}^{T}\big(\eta_{k}^{(T)}\big)^{2} & 
	\ge\frac{9\eta_{T}}{1024(1-\gamma)}.
	\label{eq:sum-eta-kT-lower-bound-567}
\end{align}



Substituting the above bound into \eqref{eq:Var-VT-2-LB-123}, we obtain
%
\begin{align}
\mathsf{Var}\big(V_{T}(2)\big) & \ge\frac{4\gamma-1}{36(1-\gamma)}\sum_{k=\tau+1}^{T}\big(\eta_{k}^{(T)}\big)^{2}\ge\frac{2}{36(1-\gamma)}\cdot\frac{9\eta_{T}}{1024(1-\gamma)} \notag\\
 & =\frac{\eta_{T}}{2048(1-\gamma)^{2}}\geq\frac{1}{4096(1-\gamma)^{4}T} , \label{eq:case2_rescaled_stepsize}
\end{align}
%
 provided that $\gamma \geq 3/4$ (so that $4\gamma - 1 \geq 2$). 
Here, the last inequality is valid since $\eta_T=\frac{1}{1+\ceta (1-\gamma) T} \geq \frac{1}{1+(1-\gamma)^2 T}\geq \frac{1}{2(1-\gamma)^2 T}$ as long as $T\geq \frac{1}{(1-\gamma)^2}$. 


\end{itemize}
%





\paragraph{Putting all this together.}

With the above bounds in place, it is readily seen that either the bias is too large (see \eqref{eq:large-tau-case-mean-diff})
%
or the variance is too large (see \eqref{eq:case2_constant_stepsize} and \eqref{eq:case2_rescaled_stepsize}). These bounds taken collectively with \eqref{eq:bias-variance-decomposition} yield
%
\begin{align}
	\mathbb{E}\big[ \big( V^{\star}(2) - V_T(2) \big)^2 \big] 
	&\geq \big( V^{\star}(2) - \mathbb{E} \big[V_T(2)\big] \big)^2 + \mathsf{Var}\big(V_{T}(2)\big)  \notag\\
	&\geq \min \left\{   \frac{1}{144(1-\gamma)^2}, \frac{1}{96(1-\gamma)^4T} , \frac{1}{4096(1-\gamma)^4T}  \right\} = \frac{1}{4096(1-\gamma)^4T} ,
	\label{eq:MSE-V2-case1}
\end{align}
%
provided $T\geq \frac{1}{(1-\gamma)^2}$.
%


\input{case3.tex}




\subsection{Proof of Lemma~\ref{lem:optimal-V-Q-hardMDP}}
\label{sec:proof-lem:optimal-V-Q-hardMDP}


Given that state $0$ is an absorbing state with zero immediate reward, it is easily seen that
%
\[
V^{\pi}(0)=0\qquad\text{for all }\pi
\qquad \Longrightarrow \qquad V^{\star}(0) = Q^{\star}(0,1)  =0.
\]
%and hence $V^{\star}(0)=0$. 
%
Moreover, by construction, taking action 1 and taking action 2 in state 1 result
in the same behavior (in terms of both the reward function and the
associated transition probability), and as a consequence,
%
\begin{equation}
Q^{\star}(1,1)=Q^{\star}(1,2)=V^{\star}(1).\label{eq:Q1-Q2-Vstar}
\end{equation}
%
From Bellman's equation, we can thus deduce that
%
\[
	Q^{\star}(1,1)=r(1,1)+\gamma P(0\mymid1,1)V^{\star}(0)+\gamma P(1\mymid1,1)V^{\star}(1),
\]
%
which in conjunction with (\ref{eq:Q1-Q2-Vstar}) and a little algebra leads to
%
\[
	V^{\star}(1)=\frac{r(1,1)+\gamma P(0\mymid1,1)V^{\star}(0)}{1-\gamma P(1\mymid1,1)}=\frac{1}{1-\gamma p}=\frac{3}{4(1-\gamma)}.
\]
%
Here, the second identity follows since $V^{\star}(0)=0$, and the
third identity makes use of \eqref{defn-parameter-p}. 
The calculation for $V^{\star}(2)$ and $Q^{\star}(2,1)$  follows from an identical argument and is hence omitted. 
%
%\end{proof}

Turning to state $3$, by Bellman's equation, we have
\[
V^{\star}(3)  = Q^{\star}(3,1)=r(3,1)+\gamma P(3\mymid 3,1)V^{\star}(3) = 1 + \gamma V^{\star}(3)  ,
\]
which leads to $V^{\star}(3) = \frac{1}{1-\gamma}$.


matrix_notation.tex
\subsection{Vector and matrix notation}
\label{subsec:matrix-notation}


%we use $\|\bm{M}\|$ and $\|\bm{M}\|_{\mathrm{F}}$ to denote the spectral norm (operator norm) and the Frobenius norm
% of $\bm{M}$, respectively. And 

To begin with, for any matrix $\bm{M}$, 
the notation $\|\bm{M}\|_{1}:=\max_{i}\sum_{j}|M_{i,j}|$
is defined as the largest row-wise $\ell_{1}$ norm of $\bm{M}$.  
For any vector $\bm{a}=[a_{i}]_{i=1}^{n}\in\mathbb{R}^{n}$, we define
$\sqrt{\cdot}$ and $|\cdot|$ in a coordinate-wise manner, i.e.~$\sqrt{\bm{a}}:=[\sqrt{a_{i}}\,]_{i=1}^{n}\in\mathbb{R}^{n}$
and $|\bm{a}|:=[|a_{i}|]_{i=1}^{n}\in\mathbb{R}^{n}$. 
For a set of vectors $\bm{a}_1,\cdots,\bm{a}_m \in \mathbb{R}^n$ with $\bm{a}_k=[a_{k,j}]_{j=1}^n$ ($1\leq k\leq m$), we define the $\max$ operator in an entrywise fashion such that 
$\max_{1\leq k\leq m} \bm{a}_k \coloneqq [ \max_{k} a_{k,j} ]_{j=1}^n$. 
For any vectors $\bm{a}=[a_{i}]_{i=1}^{n}\in\mathbb{R}^{n}$ and $\bm{b}=[b_{i}]_{i=1}^{n}\in\mathbb{R}^{n}$,
the notation $\bm{a}\leq\bm{b}$ (resp.~$\bm{a}\geq\bm{b}$) means
$a_{i}\leq b_{i}$ (resp.~$a_{i}\geq b_{i}$) for all $1\leq i\leq n$.
We also let $\bm{a}\circ\bm{b}=[a_{i}b_{i}]_{i=1}^{n}$ denote the
Hadamard product. 
In addition, we denote by $\bm{1}$ (resp.~$\bm{e}_{i}$)
the all-one vector (resp.~the $i$-th standard basis vector), and let $\bm{I}$ be the identity matrix.






We shall also introduce the matrix $\bm{P}\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|\times|\mathcal{S}|}$
to represent the probability transition kernel $P$, whose $(s,a)$-th
row $\bm{P}_{s,a}$ is a probability vector representing $P(\cdot \mymid s,a)$.
Additionally, we define the {\em square} probability transition matrix
$\bm{P}^{\pi}\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|\times|\mathcal{S}||\mathcal{A}|}$
(resp.~$\bm{P}_{\pi}\in\mathbb{R}^{|\mathcal{S}|\times|\mathcal{S}|}$)
induced by a {\em deterministic} policy $\pi$ over the state-action pairs (resp.~states) as follows:
%
\begin{equation}
	\bm{P}^{\pi}\coloneqq\bm{P}\bm{\Pi}^{\pi}\qquad\text{and}\qquad\bm{P}_{\pi}\coloneqq\bm{\Pi}^{\pi}\bm{P} ,
	\label{eqn:ppivq}
\end{equation}
%
where $\bm{\Pi}^{\pi}\in\{0,1\}^{|\mathcal{S}|\times|\mathcal{S}||\mathcal{A}|}$
is a projection matrix associated with the deterministic policy $\pi$:
%
\begin{align}
\bm{\Pi}^{\pi} & =\begin{pmatrix}\bm{e}_{\pi(1)}^{\top}\\
 & \bm{e}_{\pi(2)}^{\top}\\
 &  & \ddots\\
 &  &  & \bm{e}_{\pi(|\mathcal{S}|)}^{\top}
\end{pmatrix}\label{eqn:bigpi}
\end{align}
%
with $\bm{e}_i$ the $i$-th standard basis vector. 
Moreover, for any vector $\bm{V}\in\mathbb{R}^{|\mathcal{S}|}$,
we define $\mathsf{Var}_{\bm{P}}(\bm{V})\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$ as follows:
%
\begin{align}
	\label{eq:defn-Var-P-V}
	\mathsf{Var}_{\bm{P}}(\bm{V})=\bm{P}(\bm{V}\circ\bm{V})-(\bm{P}\bm{V})\circ(\bm{P}\bm{V}).
\end{align}
%
In other words, the $(s,a)$-th entry of $\mathsf{Var}_{\bm{P}}(\bm{V})$ corresponds to the variance $\mathsf{Var}_{s'\sim P(\cdot|s,a)}(V(s'))$ w.r.t.~the distribution  $P(\cdot\mymid s,a)$. 


Moreover, we use the vector $\bm{r}\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$
to represent the reward function $r$, so that for any $(s,a)\in\mathcal{S}\times\mathcal{A}$, the $(s,a)$-th entry of $\bm{r}$ is given by $r(s,a)$. 
Analogously, we shall employ the vectors $\bm{V}^{\pi}\in\mathbb{R}^{|\mathcal{S}|}$,
$\bm{V}^{\star}\in\mathbb{R}^{|\mathcal{S}|}$, $\bm{V}_{t}\in\mathbb{R}^{|\mathcal{S}|}$, $\bm{Q}^{\pi}\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$, $\bm{Q}^{\star}\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$ and $\bm{Q}_{t}\in\mathbb{R}^{|\mathcal{S}||\cA|}$  to represent $V^{\pi}$, $V^{\star}$, $V_t$, $Q^{\pi}$, $Q^{\star}$ and $Q_t$, respectively. 
Additionally, we define $\pi_{t}$ to be the policy associated with $Q_t$ such that for any
state-action pair $(s,a)$, 
%
\begin{align}
	\pi_{t}(s) = \min\Big\{ a' \,\big|\, Q_{t}(s,a')=\max_{a''}Q_{t}(s,a'')\Big\}.
	\label{eq:defn-pit-s}
\end{align}
%
%\begin{equation}
%\pi_{t}(a \mymid s)\coloneqq\begin{cases}
%1,\quad & \text{if }a=\min\big\{ a'\mymid Q_{t}(s,a')=\max_{a''}Q_{t}(s,a'')\big\},\\
%0, & \text{else}.
%\end{cases}\label{eq:defn-pit-infinite}
%\end{equation}
%If there is only a single action that satisfies $Q_{t}(s,a')=\max_{a''}Q_{t}(s,a'')$,
%then $\pi_{t}(a\mymid s)=1$ if $a=\arg\min_{a'}Q_{t}(s,a')$, and $\pi_{t}(a\mymid s)=0$ otherwise. 
%
In other words, for any $s\in\mathcal{S}$, the policy $\pi_{t}$
picks out the smallest indexed action that attains the largest Q-value
in the estimate $Q_{t}(s,\cdot)$. As an immediate consequence, one can easily verify 
%
\begin{equation}
\label{eq:Vt-Qt-pit-connection-infinite}
	Q_{t}\big(s,\pi_{t}(s)\big)=V_{t}(s)  \qquad \text{and} \qquad \bm{P}\bm{V}_{t}=\bm{P}^{\pi_{t}}\bm{Q}_{t}\geq\bm{P}^{\pi}\bm{Q}_{t}
\end{equation}
%
for any $\pi,$ where $\bm{P}^{\pi}$ is defined in (\ref{eqn:ppivq}). 
Further, we introduce a matrix $\bm{P}_{t}\in\{0,1\}^{|\mathcal{S}||\mathcal{A}|\times|\mathcal{S}|}$ such that
%
\begin{equation}
\bm{P}_{t}\big((s,a),s'\big):=\begin{cases}
1, & \text{if } s' = s_t(s,a)\\
0, & \text{otherwise}
\end{cases}\label{eq:defn-Pt}
\end{equation}
%
for any $(s,a)$, which is an empirical transition matrix constructed using samples collected in the $t$-th iteration. 


Finally, let $\mathcal{X}\defn \big( |\cS|, |\cA|, \frac{1}{1-\gamma}, \frac{1}{\varepsilon} \big)$. 
The notation  $f(\mathcal{X})= O(g(\mathcal{X}))$ or $f(\mathcal{X})\lesssim g(\mathcal{X})$ (resp.~$f(\mathcal{X})\gtrsim g(\mathcal{X})$) means that there exists a universal constant $C_{0}>0$ such that $|f(\mathcal{X})|\leq C_{0}|g(\mathcal{X})|$ (resp.~$|f(\mathcal{X})|\geq C_{0}|g(\mathcal{X})|$). 
The notation $f(\mathcal{X})\asymp g(\mathcal{X})$ means  $f(\mathcal{X})\lesssim g(\mathcal{X})$ and $f(\mathcal{X})\gtrsim g(\mathcal{X})$ hold simultaneously. 
We define $\widetilde{O}(\cdot)$ in the same way as $O(\cdot)$ except that it hides logarithmic factors.




% It is convenient to present our proof based on some matrix notation for MDPs.  Denoting by $\bm{e}_1,\cdots,\bm{e}_{|\mathcal{S}|}\in \mathbb{R}^{|\mathcal{S}|}$ the standard basis vectors, we can define: 
% %
% \begin{itemize}
% 	\item $\bm{r} \in \mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$: a vector representing the reward function $r$ (so that $r_{(s,a)} =r(s,a)$ for all $(s,a)\in \cS\times \cA$). 
% 	\item $\bm{V}^{\pi} \in \mathbb{R}^{|\mathcal{S}|}$: a vector representing the value function $V^{\pi}$ (so that ${V}^{\pi}_s = {V}^{\pi}(s)$ for all $s\in \cS$).
% 	\item $\bm{Q}^{\pi} \in \mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$: a vector representing the Q-function $Q^{\pi}$ (so that ${Q}^{\pi}_{(s,a)} = {Q}^{\pi}(s,a)$ for all $(s,a)\in \cS\times \cA$).
% 	\item  $\bm{V}^{\star} \in \mathbb{R}^{|\mathcal{S}|}$ and $\bm{Q}^{\star} \in \mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$: representing the optimal value function $V^{\star}$ and optimal Q-function $Q^{\star}$.
% 	\item $\bm{P}\in \mathbb{R}^{|\cS||\cA|\times |\cS|}$: a matrix representing the probability transition kernel $P$, where the $(s,a)$-th row of $\bm{P}$ is a probability vector representing $P(\cdot|s,a)$.  
% 	Denote $\bm P_{s,a}$ as the $(s,a)$-th row of the transition matrix $\bm P$. 
% 	\item $\bm{\Pi}^{\pi} \in \{0,1\}^{|\cS| \times |\cS||\cA|}$: a projection matrix associated with a given policy $\pi$ taking the following form
% \begin{align}
% \label{eqn:bigpi}
% 	\bPi^{\pi} = {\scriptsize
% 	\begin{pmatrix}
% 		\e_{\pi(1)}^\top &            &  \\
% 		           & \e_{\pi(2)}^\top &  \\
% 		           &            & \ddots \\	
% 		           &            &         & \e_{\pi(|\cS|)}^\top
% 	\end{pmatrix}  }.
% \end{align}
% 	\item $\bm{P}^{\pi} \in \mathbb{R}^{|\cS| |\cA|\times |\cS| |\cA|}$ and $\bm{P}_{\pi} \in \mathbb{R}^{|\cS| \times |\cS| }$: two {\em square} probability transition matrices induced by the policy $\pi$ over the state-action pairs and the states respectively, defined by
% 		%
% 	\begin{equation}
% 	\label{eqn:ppivq}
% 		\Pq \defn \bP \bPi^{\pi} \qquad \text{and} \qquad \Pv \defn \bPi^{\pi} \bP .
% 	\end{equation}

% \item $\bm{r}_{\pi} \in \mathbb{R}^{|\cS|}$: a reward vector restricted to the actions chosen by the policy $\pi$, namely,  $r_{\pi}(s) = r(s,\pi(s))$ for all $s\in \cS$ (or simply, $\bm{r}_{\pi}=\bm{\Pi}^{\pi}\bm{r}$). 
% \end{itemize}

preliminary-Freedman.tex

\section{Freedman's inequality}

The analysis of this work relies heavily on Freedman's inequality
\citep{freedman1975tail}, which is an extension of the Bernstein inequality
and allows one to establish concentration results for martingales.
For ease of presentation, we include a user-friendly version of Freedman's
inequality as follows.

\begin{theorem}\label{thm:Freedman}
Suppose that $Y_{n}=\sum_{k=1}^{n}X_{k}\in\mathbb{R}$,
where $\{X_{k}\}$ is a real-valued scalar sequence obeying 
\[
\left|X_{k}\right|\leq R\qquad\text{and}\qquad\mathbb{E}\left[X_{k}\mid\left\{ X_{j}\right\} _{j:j<k}\right]=0\quad\quad\quad\text{for all }k\geq1.
\]
Define
\[
W_{n}\coloneqq\sum_{k=1}^{n}\mathbb{E}_{k-1}\left[X_{k}^{2}\right],
\]
where we write $\mathbb{E}_{k-1}$ for the expectation conditional
on $\left\{ X_{j}\right\} _{j:j<k}$. Then for any given $\sigma^{2}\geq0$,
one has
\begin{equation}
\mathbb{P}\left\{ \left|Y_{n}\right|\geq\tau\text{ and }W_{n}\leq\sigma^{2}\right\} \leq2\exp\left(-\frac{\tau^{2}/2}{\sigma^{2}+R\tau/3}\right).\label{eq:Freedmans-general}
\end{equation}

In addition, suppose that $W_{n}\leq\sigma^{2}$ holds deterministically.
For any positive integer $K\geq1$, with probability at least $1-\delta$
one has
\begin{equation}
\left|Y_{n}\right|\leq\sqrt{8\max\Big\{ W_{n},\frac{\sigma^{2}}{2^{K}}\Big\}\log\frac{2K}{\delta}}+\frac{4}{3}R\log\frac{2K}{\delta}.\label{eq:Freedman-random}
\end{equation}
\end{theorem}\begin{proof}See \cite{freedman1975tail,tropp2011freedman}
for the proof of (\ref{eq:Freedmans-general}). As an immediate consequence
of (\ref{eq:Freedmans-general}), one has
\begin{equation}
\mathbb{P}\left\{ \left|Y_{n}\right|\geq\sqrt{4\sigma^{2}\log\frac{2}{\delta}}+\frac{4}{3}R\log\frac{2}{\delta}\text{ and }W_{n}\leq\sigma^{2}\right\} \leq\delta.\label{eq:Freedman-special}
\end{equation}

Next, we turn attention to (\ref{eq:Freedman-random}). Consider any
positive integer $K$. As can be easily seen, the event 
\[
\mathcal{H}_{K}
\coloneqq
\Bigg\{\left|Y_{n}\right|\ge\sqrt{8\max\Big\{ W_{n},\frac{\sigma^{2}}{2^{K}}\Big\}\log\frac{2K}{\delta}}+\frac{4}{3}R\log\frac{2K}{\delta}\Bigg\}
\]
 is contained within the union of the following $K$ events 
\[
\mathcal{H}_{K}\subseteq\bigcup_{0\leq k<K}\mathcal{B}_{k},
\]
where we define
\begin{align*}
\mathcal{B}_{k} & \coloneqq\left\{ \left|Y_{n}\right|\ge\sqrt{\frac{4\sigma^{2}}{2^{k-1}}\log\frac{2K}{\delta}}+\frac{4}{3}R\log\frac{2K}{\delta}\text{ and }\frac{\sigma^{2}}{2^{k}}\le W_{n}\leq\frac{\sigma^{2}}{2^{k-1}}\right\} ,\qquad1\leq k\leq K-1,\\
\mathcal{B}_{0} & \coloneqq\left\{ \left|Y_{n}\right|\ge\sqrt{\frac{4\sigma^{2}}{2^{K-1}}\log\frac{2K}{\delta}}+\frac{4}{3}R\log\frac{2K}{\delta}\text{ and }W_{n}\leq\frac{\sigma^{2}}{2^{K-1}}\right\} .
\end{align*}
Invoking inequality \eqref{eq:Freedman-special} with $\sigma^{2}$ set to be $\frac{\sigma^{2}}{2^{k-1}}$ and $\delta$ set to be $\frac{\delta}{K}$, 
we arrive at $\mathbb{P}\left\{ \mathcal{B}_{k}\right\} \leq\delta/K$. 
Taken this fact together with the union bound gives
\[
\mathbb{P}\left\{ \mathcal{H}_{K}\right\} \leq\sum_{k=0}^{K-1}\mathbb{P}\left\{ \mathcal{B}_{k}\right\} \leq\delta.
\]
This concludes the proof.
\end{proof}

proof_outline.tex
\subsection{Proof outline for Theorem~\ref{thm:infinite-horizon-simple}}
\label{sec:proof-outline-thm:infinite-horizon}

We are now positioned to describe how to establish Theorem \ref{thm:infinite-horizon-simple}, towards which we first express the Q-learning update rule \eqref{eqn:q-learning} and \eqref{defn:empirical-Bellman-t-inf} using the above matrix notation. 
As can be easily verified, Q-learning employs the samples in $\bm{P}_{t}$ (cf.~\eqref{eq:defn-Pt}) to perform the following update
%
%\begin{subequations}
\begin{align}
\bm{Q}_{t} & =(1-\eta_{t})\bm{Q}_{t-1}+\eta_{t}(\bm{r}+\gamma\bm{P}_{t}\bm{V}_{t-1}) \label{eq:iteration-rule-infinite}
%\label{eq:iteration-rule-infinite-Q}
%\\
%\bm{V}_{t} & =\max_{a}\bm{Q}_{t} .\label{eq:iteration-rule-infinite-V}
\end{align}
%\end{subequations} 
%
in the $t$-th iteration. In the sequel, we denote by 
%
\begin{equation}
	\bm{\Delta}_{t}\coloneqq\bm{Q}_{t}-\bm{Q}^{\star}
	\label{eq:defn-Delta-t-infinite}
\end{equation}
%
the error of the Q-function estimate in the $t$-th iteration. 



\subsubsection{Basic decomposition}

We start by decomposing the estimation error term $\bm{\Delta}_{t}$. 
In view of the update rule (\ref{eq:iteration-rule-infinite}), we
arrive at the following elementary decomposition:
%
\begin{align}
\bm{\Delta}_{t} & =\bm{Q}_{t}-\bm{Q}^{\star}=(1-\eta_{t})\bm{Q}_{t-1}+\eta_{t}\big(\bm{r}+\gamma\bm{P}_{t}\bm{V}_{t-1}\big)-\bm{Q}^{\star}\nonumber \\
 & =(1-\eta_{t})\big(\bm{Q}_{t-1}-\bm{Q}^{\star}\big)+\eta_{t}\big(\bm{r}+\gamma\bm{P}_{t}\bm{V}_{t-1}-\bm{Q}^{\star}\big)\nonumber \\
 & =(1-\eta_{t})\bm{\Delta}_{t-1}+\eta_{t}\gamma\big(\bm{P}_{t}\bm{V}_{t-1}-\bm{P}\bm{V}^{\star}\big)\nonumber \\
 & =(1-\eta_{t})\bm{\Delta}_{t-1}   +\eta_{t}\gamma\big\{\bm{P}(\bm{V}_{t-1}-\bm{V}^{\star})+(\bm{P}_{t}-\bm{P})\bm{V}_{t-1}\big\}, 
 \label{eq:iteration-infinite}
\end{align}
%
where the third line exploits the Bellman equation $\bm{Q}^{\star}= \bm{r}+ \gamma \bm{P}\bm{V}^{\star}$. 
Further, the term $\bm{P}(\bm{V}_{t-1}-\bm{V}^{\star})$ can be linked
with $\bm{\Delta}_{t-1}$ using the definition \eqref{eq:defn-pit-s} of $\pi_t$ as follows 
%
\begin{subequations}
\label{eq:V2Q-infinite}
%
\begin{align}
 \bm{P} (\bm{V}_{t-1}-\bm{V}^{\star}) & =\bm{P}^{\pi_{t-1}}\bm{Q}_{t-1}-\bm{P}^{\pi^{\star}}\bm{Q}^{\star}  \leq\bm{P}^{\pi_{t-1}}\bm{Q}_{t-1}-\bm{P}^{\pi_{t-1}}\bm{Q}^{\star}=\bm{P}^{\pi_{t-1}}\bm{\Delta}_{t-1},\\
 \bm{P}  (\bm{V}_{t-1} -\bm{V}^{\star}) &=\bm{P}^{\pi_{t-1}}\bm{Q}_{t-1}-\bm{P}^{\pi^{\star}}\bm{Q}^{\star} \geq\bm{P}^{\pi^{\star}}\bm{Q}_{t-1}-\bm{P}^{\pi^{\star}}\bm{Q}^{\star}=\bm{P}^{\pi^{\star}}\bm{\Delta}_{t-1},
\end{align}
%
\end{subequations}
%
where we have made use of the relation \eqref{eq:Vt-Qt-pit-connection-infinite}.
Substitute \eqref{eq:V2Q-infinite} into \eqref{eq:iteration-infinite} to reach
%
%\begin{subequations}
%
\begin{align}
\label{eq:Delta-onestep-UB-LB}
\begin{array}{l}
\bm{\Delta}_{t}  \le(1-\eta_{t})\bm{\Delta}_{t-1}+\eta_{t}\gamma\big\{\bm{P}^{\pi_{t-1}}\bm{\Delta}_{t-1}+(\bm{P}_{t}-\bm{P})\bm{V}_{t-1}\big\};\\
\bm{\Delta}_{t}  \geq(1-\eta_{t})\bm{\Delta}_{t-1}+\eta_{t}\gamma\big\{\bm{P}^{\pi^{\star}}\bm{\Delta}_{t-1}+(\bm{P}_{t}-\bm{P})\bm{V}_{t-1}\big\}.
\end{array}
\end{align}
%
%\end{subequations}
%
Applying these relations recursively, we obtain 
%
%\begin{subequations}
%
\begin{align}
\label{eq:Delta-allstep-UB-LB}
\begin{array}{l}
\bm{\Delta}_{t}  \le\eta_{0}^{(t)}\bm{\Delta}_{0}+\sum\limits_{i=1}^{t}\eta_{i}^{(t)}\gamma\big\{\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}+(\bm{P}_{i}-\bm{P})\bm{V}_{i-1}\big\}, \\
\bm{\Delta}_{t}  \ge\eta_{0}^{(t)}\bm{\Delta}_{0}+\sum\limits_{i=1}^{t}\eta_{i}^{(t)}\gamma\big\{\bm{P}^{\pi^{\star}}\bm{\Delta}_{i-1}+(\bm{P}_{i}-\bm{P})\bm{V}_{i-1}\big\}, 
\end{array}
\end{align}
%
%\end{subequations}
%
where we define
%
\begin{equation}
\eta_{i}^{(t)}\coloneqq\begin{cases}
\prod_{j=1}^{t}(1-\eta_{j}), & \text{if }i=0,\\
\eta_{i}\prod_{j=i+1}^{t}(1-\eta_{j}), & \text{if }0<i<t,\\
\eta_{t}, & \text{if }i=t.
\end{cases}\label{def:eta-i-t-simple}
\end{equation}


\paragraph{Comparisons to prior approaches.} 

We take a moment to discuss how prior analyses handle the above elementary decomposition. 
Several prior works (e.g., \citet{wainwright2019stochastic,li2020sample}) tackled the second term on the right-hand side of the relation \eqref{eq:Delta-onestep-UB-LB} via the following crude bounds:
%
\begin{align*} 
	\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}  &\leq \big\|\bm{P}^{\pi_{i-1}} \big\|_1  \| \bm{\Delta}_{i-1} \|_{\infty} \bm{1} = \| \bm{\Delta}_{i-1} \|_{\infty} \bm{1} , \\
	 \bm{P}^{\pi^{\star}}\bm{\Delta}_{i-1}  &\geq - \big\|\bm{P}^{\pi^{\star}} \big\|_1  \| \bm{\Delta}_{i-1} \|_{\infty} \bm{1} = - \| \bm{\Delta}_{i-1} \|_{\infty} \bm{1} ,
\end{align*}
%
which, however, are too loose when characterizing the dependency on $\frac{1}{1-\gamma}$. 
By contrast, expanding terms recursively without the above type of crude bounding and carefully analyzing the aggregate terms (e.g., $\sum_{i=1}^t \eta_{i}^{(t)}\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}$) play a major role in sharpening the dependence of sample complexity on the effective horizon.   


\subsubsection{Key intertwined relations underlying $\{\|\bm{\Delta}_{t}\|_{\infty}\}$}


By exploiting the crucial relations \eqref{eq:Delta-allstep-UB-LB} derived above, we proceed to upper and lower bound $\bm{\Delta}_{t}$ separately. 
To be more specific, defining 
%
\begin{align}
	\beta\coloneqq\frac{c_{4}(1-\gamma)}{\log T}
\end{align}
%
for some constant $c_4>0$, one can further decompose the upper bound in \eqref{eq:Delta-allstep-UB-LB} into several terms:
%
\begin{align}
	\bm{\Delta}_{t}  & \leq\underbrace{\eta_{0}^{(t)}\bm{\Delta}_{0}+\sum_{i=1}^{(1-\beta)t}   \eta_{i}^{(t)}\gamma\big(\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}+(\bm{P}_{i}-\bm{P})\bm{V}_{i-1}\big)}_{=:\,\bm{\zeta}_{t}} \\
	& \qquad \qquad +    \underbrace{\sum_{i=(1-\beta)t+1}^{t}  \eta_{i}^{(t)}\gamma(\bm{P}_{i}-\bm{P})\bm{V}_{i-1}}_{=:\,\bm{\xi}_{t}}  	+\sum_{i=(1-\beta)t+1}^{t}  \eta_{i}^{(t)}  \gamma\bm{P}^{\pi_{i-1}}\bm{\Delta}_{i-1}.  
	\label{eq:Delta-UB2}
\end{align}
%
Let us briefly remark on the effect of the first two terms:
%
\begin{itemize}
	\item Each component in the first term $\bm{\zeta}_t$ is fairly small, given that $\eta_{i}^{(t)}$ is sufficiently small for any $i\leq (1-\beta)t$ (meaning that each component has undergone contraction --- the ones taking the form of $1-\eta_j$ --- for sufficiently many times). As a result, the influence of $\bm{\zeta}_t$ becomes somewhat negligible. 

	\item The second term $\bm{\xi}_t$, which can be controlled via Freedman's inequality \citep{freedman1975tail} due to its martingale structure, contributes to the main variance term in the above recursion.  Note, however, that the resulting variance term also depends on $\{\bm{\Delta}_i\}$.
		%, which eventually allows for a recurrence relation stated in terms of $\{\bm{\Delta}_i\}$. 
	
\end{itemize}
%
In summary, the right-hand side of the above inequality can be further decomposed into some weighted superposition of $\{\bm{\Delta}_i\}$ in addition to  some negligible effect. This is formalized in the following two lemmas, which make apparent the key intertwined relations underlying $\{\bm{\Delta}_i\}$.
%
%\footnote{Here and throughout, the notation $f(n)\lesssim g(n)$ (resp.~$f(n)\gtrsim g(n)$) means that there exists a constant $C_{0}>0$ such that $|f(n)|\leq C_{0}|g(n)|$ (resp.~$|f(n)|\geq C_{0}|g(n)|$).}
%
\begin{lemma}
\label{lem:inf-upper-bound}
	Suppose that $c_1c_2\leq c_4/8$. 
	%There exists some universal constant $c_{\mathrm{b}}>0$ such that 
	With probability at least $1-\delta$,
 %
\begin{align*}
\bm{\Delta}_{t} & \leq 30 \sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\Big(1+\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\Big)}\ \bm{1} 
\end{align*}
holds simultaneously for all $t\ge\frac{T}{c_{2}\log T}$.
\end{lemma}
%
% Similarly, making use of the lower bound in \eqref{eq:Delta-allstep-UB-LB} allows one to develop an analogous lower bound as follows.
%
\begin{lemma}
\label{lem:inf-lower-bound}
%
Suppose that $c_1c_2\leq c_4/8$. 
%There exists some universal constant $c_{\mathrm{b}}>0$ such that 
With probability at least $1-\delta$, 
%
\begin{equation*}
\bm{\Delta}_{t}\geq - 30 \sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\Big(1+\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\Big)}\ \bm{1} 
\end{equation*}
%
holds simultaneously for all $t\ge\frac{T}{c_{2}\log T}$. 
%
\end{lemma}
%
\begin{proof}
	The proofs of Lemma~\ref{lem:inf-upper-bound} and Lemma~\ref{lem:inf-lower-bound} are deferred to  Appendices~\ref{sec:proof_inf_upper_bound} and \ref{sec:proof_inf_lower_bound}, respectively. As a remark, our analysis collects all the error terms accrued through the iterations --- instead of bounding them individually --- by conducting a high-order nonlinear expansion of the estimation error through recursion, followed by careful control of the main variance term leveraging the structure of the discounted MDP.  
\end{proof}
 
Putting the preceding bounds in Lemmas~\ref{lem:inf-upper-bound} and \ref{lem:inf-lower-bound}
together, we arrive at
%
\begin{align}
\|\bm{\Delta}_{t}\|_{\infty} & \leq 30 \sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\gamma^2(1-\gamma)^{4}T}\bigg(1+\max_{\frac{t}{2}\le i<t}\|\bm{\Delta}_{i}\|_{\infty}\bigg)}
\label{eq:Delta-t-inf-recursion-inf-outline}
\end{align}
%
for all  $t\ge\frac{T}{c_{2}\log T}$ 
with probability exceeding $1-2\delta$, which forms the crux of our analysis. 
Employing elementary analysis tailored to the above recursive relation, one can demonstrate that
 %
\begin{align}
	\|\bm{\Delta}_{T}\|_{\infty} \leq O\bigg( \sqrt{\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{4}T}} +\frac{\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{4}T} 
	\bigg) \label{eq:Delta-T-UB-final-inf}
\end{align}
%
with probability at least $1-2\delta$, which in turn allows us to establish the advertised result under the assumed sample size condition. The details are deferred to Appendix~\ref{sec:combine_ub_lb}. 


 
 \subsection{Proof outline for Theorem~\ref{thm:LB-example}}
 \label{sec:MDP-construction-hard}
 
 \paragraph{Construction of a hard instance and its property}
Let us construct an MDP $\mathcal{M}_{\mathsf{hard}}$ with state space $\cS=\{0,1,2,3\}$ (see a pictorial illustration in Figure~\ref{fig:hard-MDP}). We shall denote by $\cA_s$ the action space associated with state $s$. 
The probability transition kernel and reward function of $\mathcal{M}_{\mathsf{hard}}$ are specified as follows 
%
\begin{subequations}
\label{eq:construction-hard-MDP}
\begin{align}
	& \mathcal{A}_0=\{1\},  & P(0 \mymid 0, 1) &= 1, &\quad && r(0, 1) = 0, \\
	& \mathcal{A}_1=\{1,2\},  & P(1 \mymid 1, 1) &= p,   & P(0 \mymid 1, 1) = 1-p,   && r(1, 1) = 1,  \\
	& \quad   & P(1 \mymid 1, 2) &= p,  & P(0 \mymid 1, 2) = 1-p,  && r(1, 2) = 1, \\
	& \mathcal{A}_2=\{1\},  & P(2 \mymid 2, 1) &= p,  & P(0 \mymid 2,1) = 1-p,  && r(2,1) = 1, \\
	& \mathcal{A}_3=\{1\},  & P(3 \mymid 3, 1) &= 1,  &\quad && r(3,1) = 1, 
\end{align}
\end{subequations}
%
where the parameter $p$  is taken to be 
%
\begin{align}
	\label{defn-parameter-p}
	p = \frac{4\gamma-1}{3\gamma} .
\end{align}
%
% and
%\begin{align*}
%r(0) = 0, \qquad r(1, 1) = r(1, 2) = 1, \qquad r(2) = 1.
%\end{align*}
\begin{figure}[t]
	\centering{\includegraphics[width=0.6\textwidth]{figures/hard_MDP_example.pdf}}
	\label{fig:hard-MDP}
	\caption{The constructed hard MDP instance used in the analysis of Theorem~\ref{thm:LB-example}, where $p= \frac{4\gamma-1}{3\gamma}$ and the specifications are described in \eqref{eq:construction-hard-MDP}.}
\end{figure}


Before moving forward to analyze the behavior of Q-learning, we first characterize the optimal value function and Q-function of this MDP; the proof is postponed to Section~\ref{sec:proof-lem:optimal-V-Q-hardMDP}. 
%
\begin{lemma}
\label{lem:optimal-V-Q-hardMDP}
%
Consider the MDP $\mathcal{M}_{\mathsf{hard}}$ constructed in \eqref{eq:construction-hard-MDP}. One has
%
\begin{subequations}
\label{eq:optimal-V-Q-hardMDP}
\begin{align}
&V^{\star}(0) = Q^{\star}(0,1) = 0; \\
&V^{\star}(1) = Q^{\star}(1, 1) = Q^{\star}(1, 2) = V^{\star}(2) = Q^{\star}(2,1) = \frac{1}{1-\gamma p}= \frac{3}{4(1-\gamma)}; \\
&V^{\star}(3) = Q^{\star}(3, 1) = \frac{1}{1-\gamma}.
\end{align}
\end{subequations}
%
\end{lemma}

Recognizing the elementary decomposition
%
\begin{equation}
	\mathbb{E}\left[\big(V^{\star}(s)-V_{T}(s)\big)^{2}\right]=\big(\mathbb{E}\left[V^{\star}(s)-V_{T}(s)\right]\big)^{2}+\mathsf{Var}\big(V_{T}(s)\big)
	\label{eq:bias-variance-decomposition}
\end{equation}
%
for any state $s$, our proof consists of lower bounding either the squared bias term $\big( \mathbb{E}[V^{\star}(s)-V_{T}(s)] \big)^2$ or the variance term $\mathsf{Var}\big(V_{T}(s)\big)$. 
In short,  we shall primarily analyze the dynamics w.r.t.~state 2 to handle the case when the learning rates are either too small or too large, and analyze the dynamics w.r.t.~state 1 to cope with the case with medium learning rates (with state 3 serving as a helper state to simplify the analysis). The latter case --- corresponding to the learning rates adopted in establishing the upper bounds --- is the most challenging: critically, from state 1 the agent can take one of two identical actions, whose value tends to be estimated with a high positive bias due to maximizing over the empirical state-action values, highlighting the well-recognized ``over-estimation'' issue of Q-learning  in practice \citep{hasselt2010double}.  The complete proof is deferred to Appendix~\ref{sec:lower-bounds}.
 


results.tex
\section{Main results: sample complexity of synchronous Q-learning}
\label{sec:main-results}

With the above backgrounds in place, we are in a position to state formally our main findings in this section, concentrating on the synchronous setting.  




\subsection{Minimax optimality of TD learning}



We start with the special with $|\cA|=1$ and characterize the $\ell_{\infty}$-based sample complexity of synchronous TD learning. 


\begin{theorem}
\label{thm:policy-evaluation}
	Consider any $\delta\in(0,1)$,  $\varepsilon\in(0,1]$, and $\gamma\in [1/2,1)$. Suppose that for any $0\leq t\leq T$, the
learning rates satisfy
%
\begin{subequations}
\label{eq:thm-td}
%
\begin{equation}
\frac{1}{1+\frac{c_1 (1-\gamma) T}{\log^2 T}}\le\eta_{t}\le\frac{1}{1+\frac{c_2 (1-\gamma) t}{\log^2 T}} \label{eq:thm:eta-TD}
\end{equation}
%
for some small enough universal constants $c_{1}\geq c_{2}>0$. 
Assume that the total number of iterations $T$ obeys
%
\begin{equation}
	T\ge\frac{c_{3}\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}|T}{\delta}\big)}{(1-\gamma)^{3}\varepsilon^{2}}
	\label{eq:thm:sample-size-TD-thm}
\end{equation}
%
for some sufficiently large universal constant $c_{3}>0$. 
\end{subequations}
%
If the initialization obeys $0\leq {V}_{0}(s) \leq\frac{1}{1-\gamma}$ for all $s\in \cS$, 
then  with probability at least $1-\delta$, Algorithm~\ref{alg:td-infinite} achieves
%
\[
	\max_{s\in \cS} \big| {V}_{T}(s) -  {V}^{\star}(s) \big| \le\varepsilon.
\]
% 
\end{theorem}
%
\begin{remark} \label{remark:expected-error-inf-td}
	This high-probability bound immediately translates to a mean estimation error guarantee. Recognizing the crude upper bound 
	$\big| {V}_{T}(s) -  {V}^{\star}(s) \big| \leq \frac{1}{1-\gamma}$ (see \eqref{eq:V-i-LB-UB} in Section~\ref{sec:infinite_prelim_TD}) and taking $\delta\leq {\varepsilon(1-\gamma)}$, we reach
	%
	\begin{equation}
		\mathbb{E}\Big[ \max_{s} \big| {V}_{T}(s) -  {V}^{\star}(s) \big| \Big] \le \varepsilon (1-\delta)+\delta \frac{1}{1-\gamma}
		\leq 2\varepsilon,
	\end{equation}
	%
	provided that $T\ge\frac{c_{3} (\log^{3}T)\big(\log\frac{|\mathcal{S}|T}{\varepsilon (1-\gamma)}\big)}{(1-\gamma)^{3}\varepsilon^{2}}$. 
\end{remark}
%



Given that each iteration of synchronous TD learning makes use of $|\cS|$ samples, 
Theorem~\ref{thm:policy-evaluation} implies that the sample complexity of TD learning is at most
%
\begin{equation}
	\label{eq:TD-sample-complexity-simple}
	\widetilde{O} \bigg( \frac{|\cS|}{(1-\gamma)^3 \varepsilon^2} \bigg)
\end{equation}
%
for any target accuracy level $\varepsilon \in (0,1]$. 
This non-asymptotic result is valid as long as
the learning rates are chosen to be either a proper constant or rescaled linear (see~\eqref{eq:thm:eta-TD}).  
Compared to a large number of prior works studying the performance of TD learning \citep{borkar2000ode,bhandari2021finite,khamaru2021temporal,wainwright2019stochastic,chen2020finite,lakshminarayanan2018linear}, 
 Theorem~\ref{thm:policy-evaluation} strengthens prior results by uncovering an improved scaling (i.e., $\frac{1}{(1-\gamma)^3}$) in the effective horizon. In fact, prior results on plain TD learning were only able to obtain a scaling as $\frac{1}{(1-\gamma)^5}$ \citep{wainwright2019stochastic}.    


To assess the tightness of the above result, we take a moment to compare it with the minimax lower bound recently established in the context of value function estimation. Specifically,  
\citet[Theorem 2(b)]{pananjady2019value}
asserted that no algorithm whatsoever can obtain an entrywise $\varepsilon$ approximation of the value function---in a minimax sense---unless the total sample size exceeds
%
\begin{equation}
	\label{eq:TD-sample-complexity-simple-LB}
	\widetilde{\Omega} \bigg( \frac{|\cS|}{(1-\gamma)^3 \varepsilon^2} \bigg). 
\end{equation}
%
In turn, this taken together with Theorem~\ref{thm:policy-evaluation} unveils the minimax optimality of the sample complexity (modulo some logarithmic factor) of TD learning  for the synchronous setting.  
While prior works have demonstrated how to attain the minimax limit \eqref{eq:TD-sample-complexity-simple-LB} using model-based methods or variance-reduced model-free algorithms (e.g., \citet{azar2013minimax,pananjady2019value,li2020breaking,khamaru2021temporal}), 
our theory provides the first rigorous evidence that plain TD learning alone is already minimax optimal, without the need of Polyak-Ruppert averaging or variance reduction. 



%\paragraph{Minimax optimality.} 



%
%\paragraph{Learning rates.} In view of the assumption \eqref{eq:thm:infinite-horizaon-eta}, our result accommodates two commonly adopted learning rate schemes: (i) linearly rescaled learning rates $\frac{1}{1+\frac{c_2 (1-\gamma)}{\log^2 T}t}$, and (ii) iteration-invariant learning rates $\frac{1}{1+\frac{c_1 (1-\gamma) T}{\log^2 T}}$ (which depend on the total number of iterations $T$ but not the iteration number $t$).
%In particular, when the sample size is $T=\frac{c_{3} (\log^{4}T)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{4}\varepsilon^{2}}$, the constant learning rates can be taken to be on the order of
%%
%\[
%	\eta_t \equiv \widetilde{O} \big( (1-\gamma)^3 \varepsilon^2 \big),  \qquad 0\leq t\leq T, 
%\]
%%
%which depends almost solely on the discount factor $\gamma$ and the target accuracy $\varepsilon$. 
%Interestingly, both learning rate schedules lead to the same $\ell_{\infty}$-based sample complexity bound (in an order-wise sense), making them appealing for practical use.  




\subsection{Tight sample complexity and sub-optimality of Q-learning}
%\label{sec:main-results-inf}

 
Next, we move on to the more general case with $|\cA|\geq 2$ and study the performance of Q-learning. 
As it turns out, Q-learning with $|\cA|\geq 2$ is considerably more challenging to analyze than the TD learning case,
due to the presence of the nonsmooth max operator.  
Our $\ell_{\infty}$-based sample complexity bound for Q-learning is summarized as follows, 
strengthening the state-of-the-art results. 
%
\begin{theorem}
\label{thm:infinite-horizon-simple}
%
Consider any $\delta\in(0,1)$,  $\varepsilon\in(0,1]$, and $\gamma\in [1/2,1)$. Suppose that for any $0\leq t\leq T$, the
learning rates satisfy
%
\begin{subequations}
\label{eq:thm-infinite-horizon-condition}
%
\begin{equation}
\frac{1}{1+\frac{c_1 (1-\gamma) T}{\log^3 T}}\le\eta_{t}\le\frac{1}{1+\frac{c_2 (1-\gamma) t}{\log^3 T}} \label{eq:thm:infinite-horizaon-eta}
\end{equation}
%
for some small enough universal constants $c_{1}\geq c_{2}>0$. 
Assume that the total number of iterations $T$ obeys
%
\begin{equation}
	T\ge\frac{c_{3}\big(\log^{4}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{4}\varepsilon^{2}}
	\label{eq:thm:infinite-horizon-T}
\end{equation}
%
for some sufficiently large universal constant $c_{3}>0$. 
\end{subequations}
%
If the initialization obeys $0\leq {Q}_{0}(s,a) \leq\frac{1}{1-\gamma}$ for any $(s,a)\in \cS\times \cA$, 
then Algorithm \ref{alg:q-infinite} achieves
%
\[
	\max_{(s,a)\in \cS\times \cA} \big| {Q}_{T}(s,a) -  {Q}^{\star}(s,a) \big| \le\varepsilon
\]
%
 with probability at least $1-\delta$. 
\end{theorem}
%
\begin{remark} \label{remark:expected-error-inf}
	Repeating exactly the same argument as in Remark~\ref{remark:expected-error-inf-td}, one can readily translate this 
high-probability bound into the following  mean estimation error guarantee: 
	%
	\begin{equation}
		\mathbb{E}\Big[ \max_{s,a} \big| {Q}_{T}(s,a) -  {Q}^{\star}(s,a) \big| \Big] \le \varepsilon (1-\delta)+\delta \frac{1}{1-\gamma}
		\leq 2\varepsilon,
	\end{equation}
	%
	holds as long as $T\ge\frac{c_{3} (\log^{4}T)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\varepsilon (1-\gamma)}\big)}{(1-\gamma)^{4}\varepsilon^{2}}$. 
\end{remark}
%


In a nutshell, Theorem~\ref{thm:infinite-horizon-simple} develops a non-asymptotic bound on the iteration complexity of Q-learning in the presence of the synchronous model.  A few remarks and implications are in order. 


\paragraph{Sample complexity and sharpened dependency on $\frac{1}{1-\gamma}$.} 
%
Recognizing that $|\cS||\cA|$ independent samples are drawn in each iteration, 
we can see from Theorem~\ref{thm:infinite-horizon-simple} the following sample complexity bound
%
\begin{equation}
	\widetilde{O}\Big( \frac{|\cS||\cA|}{(1-\gamma)^4 \varepsilon^2} \Big)
	\label{eq:sample-complexity-sync-Q-learning}
\end{equation}
%
in order for Q-learning to attain $\varepsilon$-accuracy ($0<\varepsilon<1$) in an entrywise sense.  
To the best of our knowledge, this is the first result that breaks the $\frac{|\cS||\cA|}{(1-\gamma)^5 \varepsilon^2}$ barrier that is present in all state-of-the-art analyses for vanilla Q-learning \citep{beck2012error,wainwright2019stochastic,chen2020finite,qu2020finite,li2020sample}. 
%In addition, the dependence of our result on the effective horizon (i.e., $\frac{1}{(1-\gamma)^4}$) matches the scaling of a numerical example exhibited in \citet[Figure~2]{wainwright2019stochastic}. 



\paragraph{Learning rates.} Akin to the TD learning case, 
our result accommodates two commonly adopted learning rate schemes (cf.~\eqref{eq:thm:infinite-horizaon-eta}): (i) linearly rescaled learning rates $\frac{1}{1+\frac{c_2 (1-\gamma)}{\log^2 T}t}$, and (ii) iteration-invariant learning rates $\frac{1}{1+\frac{c_1 (1-\gamma) T}{\log^2 T}}$ (which depend on the total number of iterations $T$ but not the iteration number $t$).
In particular, when the sample size is $T=\frac{c_{3} (\log^{4}T)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{(1-\gamma)^{4}\varepsilon^{2}}$, the constant learning rates can be taken to be on the order of
%
\[
	\eta_t \equiv \widetilde{O} \big( (1-\gamma)^3 \varepsilon^2 \big),  \qquad 0\leq t\leq T, 
\]
%
which depends almost solely on the discount factor $\gamma$ and the target accuracy $\varepsilon$. 
Interestingly, both learning rate schedules lead to the same $\ell_{\infty}$-based sample complexity bound (in an order-wise sense), making them appealing for practical use.  



%\subsection{A matching lower bound for Q-learning}


\paragraph{A matching lower bound and sub-optimality.} 

The careful reader might remark that there remains a gap between our sample complexity bound for Q-learning and the minimax lower bound \citep{azar2013minimax}. More specifically, the minimax lower bound scales on the order of $\frac{|\cS||\cA|}{(1-\gamma)^3\varepsilon^2}$ and is achievable --- up to some logarithmic factor --- by the model-based approach and variance-reduced methods \citep{azar2013minimax,agarwal2019optimality,li2020breaking,wainwright2019variance}. 
This raises natural questions regarding whether our sample complexity bound can be further improved, and whether there is any intrinsic bottleneck that prevents vanilla Q-learning from attaining optimal performance.  
To answer these questions, we develop the following algorithm-dependent lower bound, 
which confirms the sharpness of Theorem~\ref{thm:infinite-horizon-simple} and reveals the sub-optimality of vanilla Q-learning.
%
%or $T \ge \frac{100}{\ceta(1-\gamma)}$
\begin{theorem}
\label{thm:LB-example}
	Assume that $3/4\leq \gamma < 1$ and that $T\geq \frac{c_3}{(1-\gamma)^2}$ for some sufficiently large constant $c_3>0$. 
	Suppose that the initialization is $Q_0 \equiv 0$, and that the learning rates are taken to be either (i) $\eta_t = \frac{1}{1+\ceta (1-\gamma)t}$ for all $t\geq 0$, or (ii) $\eta_t \equiv \eta $ for all $t\geq 0$. 
	There exists a $\gamma$-discounted MDP with $|\cS|=4$ and $|\cA|=2$ such that Algorithm \ref{alg:q-infinite} --- with any $\ceta >0$ and any $\eta \in (0,1)$ --- obeys
\[
	\max_s \mathbb{E} \Big[ \big|V_T(s) - V^{\star}(s)\big|^2 \Big] 
	\geq  \frac{c_{\mathsf{lb}}}{(1-\gamma)^4T\log^2 T} ,
\] 
where $c_{\mathsf{lb}} > 0$ is some universal constant.
\end{theorem}
%
As asserted by this theorem, it is impossible for Q-learning to attain $\varepsilon$-accuracy (in the sense that $\max_s \mathbb{E} \big[ \big|V_T(s) - V^{\star}(s)\big|^2 \big] \leq \varepsilon^2$) unless the number of iterations exceeds the order of 
%
\[
	\frac{1}{(1-\gamma)^{4}\varepsilon^{2}}
\]
%
up to some logarithmic factor. Consequently, the upper bound in Theorem~\ref{thm:infinite-horizon-simple} is tight in terms of its dependency on the effective horizon $\frac{1}{1-\gamma}$, which is larger than the minimax limit \citep{azar2013minimax} by a factor of $\frac{1}{1-\gamma}$. 
As a consequence, we need to resort to more sophisticated tricks like variance reduction in order to attain minimax optimality when $|\cA|\geq 2$ \citep{wainwright2019variance,li2020sample}. 



finite-horizon-write-ups.tex

\subsection{Q-learning for finite-horizon MDPs} 
\label{sec:background-finite}


\paragraph{Finite-horizon MDPs.} 

We now turn attention to a finite-horizon MDP, which can be represented and described by the quintuple $\mathcal{M} = (\mathcal{S}, \mathcal{A}, \{P_h\}_{h=1}^H, \{r_h\}_{h=1}^H, H)$. Here, $H$ represents the time horizon of the MDP. For any $1\leq h\leq H$, we let $P_h : \cS \times \cA \rightarrow \Delta(\cS)$ denote the probability transition kernel at step $h$ (i.e., $P_h(s'\mymid s,a)$ is the probability of transiting to $s'$ from $(s,a)$ at step $h$), 
and $r_h: \cS \times \cA \rightarrow [0,1]$ indicates the reward function at step $h$ (i.e., $r_h(s,a)$ is the immediate reward at step $h$ in response to $(s,a)$). As before, we assume normalized rewards so that all the $r_h(s,a)$'s reside within the range $[0,1]$. 
%In contrast to the infinite-horizon counterpart, finite-horizon MDPs do not require introducing a discount factor $\gamma$.



\paragraph{Value function and Q-function.}

Let  $\pi=\{\pi_h\}_{1\leq h\leq H}$ represent a deterministic policy such that $\pi_h(s)$ specifies the action selected at step $h$ when the current state is $s$. The value function and Q-function associated with policy $\pi$ are defined by
%
 \begin{align*}
   V^{\pi}_h (s) \coloneqq \mathbb{E} \left[ \sum_{k=h}^{H} r_k(s_k,a_k ) \,\Big|\, s_h =s \right]
\end{align*} 
%
 for all $s\in \cS$ and all $1\leq h \leq H$, and
 %
 \begin{equation*}
Q^{\pi}_h(s,a) \coloneqq \mathbb{E} \left[ \sum_{k=h}^{H} r_k(s_k,a_k ) \,\Big|\, s_h =s, a_h = a \right] 
\end{equation*} 
%
for all $(s,a)\in \cS \times \cA$  and all $1\leq h \leq H$. 
As before,  the expectations are taken over the randomness of the MDP trajectory $\{(s_k,a_k)\}_{1\leq k\leq H}$ induced by the transition kernel $\{ P_h \}_{h=1}^H$ when policy $\pi$ is adopted.
In view of the assumed bounds for the immediate rewards, it is easily seen that $0\leq V^{\pi}_h (s) \leq H$ and $0 \leq Q^{\pi}_h (s, a) \leq H$ for any $\pi$, any state-action pair $(s, a)$, and any step $h$. 
Akin to the infinite-horizon scenario, the optimal value functions $\{V_h^{\star}\}_{1\leq h\leq H}$ and optimal Q-functions $\{Q_h^{\star}\}_{1\leq h\leq H}$ are defined respectively by
%
\begin{equation*}
	V^{\star}_h(s) \coloneqq \max_{\pi} V^{\pi}_h(s)
	\qquad \text{and} \qquad
	Q^{\star}_h(s,a) \coloneqq \max_{\pi} Q^{\pi}_h(s,a)
\end{equation*}
%
for any state-action pair $(s,a)\in \cS\times \cA$ and any $1\leq h \leq H$.
% A distinguishing feature of finite-horizon MDPs is the non-stationarity of  value functions and Q-functions across $h$. 


\begin{algorithm}[t]
	\begin{algorithmic}[1] 
		\STATE\textbf{inputs:} learning rates $\{\eta_t\}$, number of iterations $T$, initial estimate $\{Q_{0, h}\}_{h=1}^H$.
%\STATE  \textbf{initialization:} $Q_0=0$. 
   \FOR{$t=1,2,\cdots,T$}
   \STATE{Set $Q_{t, H+1} = 0$.}
   \FOR{$h=H, H-1, \cdots,1$}
		\STATE{ Draw  $s_{t,h}(s,a) \sim  P_h(\cdot \mymid s,a)$ for each $(s,a)$. }
	\STATE{ Compute $Q_t$ according to \eqref{eqn:q-learning-finite} and \eqref{defn:empirical-Bellman-t-finite}.}
	\ENDFOR
\ENDFOR
   %\textbf{return:} ${Q}_{T}$. 
   \end{algorithmic} 
    \caption{Synchronous Q-learning for finite-horizon MDPs}
 \label{alg:q-finite}
\end{algorithm}

\paragraph{Q-learning.}

Assume that we have access to a generative model; in each iteration, the generative model produces an independent sample for each triple $(s,a,h)$ as follows
%
\[
	s_{t,h}(s,a) \sim P_h(\cdot \mymid s,a).
\]
%
In the $t$-th iteration,  
the synchronous Q-learning algorithm updates the Q-function estimate $Q_{t,h}: \cS \times \cA \rightarrow \mathbb{R}$ from $h = H$ to $1$ as follows:
%
\begin{equation}
\label{eqn:q-learning-finite}
	Q_{t,h}  = (1- \eta_t ) Q_{t-1,h} + \eta_t \mathcal{T}_{t,h} (Q_{t, h+1}),
	%\qquad H \geq h \geq 1. 
\end{equation}
%
where $\eta_t$ denotes the learning rate in the $t$-th iteration, and $\mathcal{T}_{t,h}$ is the empirical Bellman operator based on samples generated in the $t$-th iteration for step $h$, namely,
%
\begin{align} \label{defn:empirical-Bellman-t-finite}
	\mathcal{T}_{t,h} (Q) (s , a ) &\coloneqq 
		 r_h(s , a ) + \max_{a' \in \cA} Q(s_{t,h}, a'),   \qquad	 s_{t, h}  \equiv s_{t, h}(s,a) \sim P_h(\cdot \mymid s,a)  
\end{align}
%
for every state-action pair $(s,a) \in \cS \times \cA$. 
Here, $\mathcal{T}_{t,h}$ can also be viewed as an unbiased estimate of the Bellman operator defined for the finite-horizon case. 


In addition, the initialization $\{Q_{0, h}\}_{h=1}^H$ of the algorithm is chosen such that $0 \leq Q_{0, h}(s, a) \leq H-h+1$ for all $(s,a)$ and all $1\leq h\leq H$. Throughout this paper, the value function estimate $V_{t,h}: \cS \rightarrow \mathbb{R}$ associated with the $t$-th iterate and step $h$ is defined as
%
\begin{align}
	\label{defn:Vt-finite}
	\forall s\in \cS, ~~h\in \{1,\cdots,H\}: \qquad V_{t,h}(s) :=  \max_{a\in \cA} Q_{t,h}(s,a) .
\end{align}
%
The complete algorithm is summarized in Algorithm~\ref{alg:q-finite}.



\begin{remark}
Suppose that the probability transition kernel of the MDP is time-invariant, namely,  $P_h\equiv P$ for all $1\leq h\leq H$.
Then we only need to sample once for each $(s,a)$ --- with a total number of $|\cS||\cA|$ samples --- in each iteration $t$. This should be contrasted with the time-varying case where a total number of $|\cS||\cA|H$ samples are generated in each iteration. Both scenarios have been studied for the finite-horizon setting; see, e.g., \citet{jin2018q,sidford2018near}.
\end{remark}


\subsection{Performance guarantees: finite-horizon MDPs}
\label{sec:main-results-finite}


%At each time $t$, given the samples $\bm{P}_{t,h}$, the Q-learning
%algorithm adopts the following iterative update rule
%\begin{subequations}
%\begin{align}
%\bm{Q}_{t,h} & =(1-\eta_{t})\bm{Q}_{t-1,h}+\eta_{t}(\bm{r}_{h}+\bm{P}_{t,h}\bm{V}_{t,h+1}), \forall1\le h\le H,\\
%\bm{V}_{t,h} & =\max_{a}\bm{Q}_{t,h},\qquad\forall1\le h\le H,
%\end{align}
%\end{subequations} where $\eta_{t}>0$ denotes the learning rate
%and $\bm{Q}_{t,H+1}=\bm{0},\bm{V}_{t,H+1}=\bm{0}$.



Next, we move forward to present an analogous $\ell_{\infty}$-based sample complexity of Q-learning for finite-horizon MDPs --- the setting formulated in Section~\ref{sec:background-finite}. 
%
\begin{theorem}
\label{thm:finite-horizon-simple}
Consider an arbitrary
quantity $\delta\in(0,1)$ and $\varepsilon\in(0,1]$. Suppose that
the learning rates obey
%
\begin{subequations}
\label{eq:thm-finite-horizon-condition}
%
\begin{equation}
	\frac{1}{1+\frac{c_{1}T}{H\log^{2}T}}\le\eta_{t}\le\frac{1}{1+\frac{c_{2}t}{H\log^{2}T}},\quad0\leq t\leq T
	\label{eq:thm:finite-horizaon-eta}
\end{equation}
%
for some small enough universal constants $c_{1}\geq c_{2}>0$, and
assume that the total number of iterations $T$ obeys
%
\begin{equation}
	T\ge\frac{c_{3}H^{4}\big(\log^{3}T\big)\big(\log\frac{|\mathcal{S}||\mathcal{A}|T}{\delta}\big)}{\varepsilon^{2}}
	\label{eq:thm:finite-horizon-T}
\end{equation}
%
for some sufficiently large universal constant $c_{3}>0$. \end{subequations}
If the initialization obeys $0\leq Q_{0,h}(s,a) \leq H+1-h$ for any $(s,a)$ and any $1\le h\le H$,
then Algorithm~\ref{alg:q-finite} achieves
%
\[
	\max_{(s,a)\in \cS\times \cA, \, 1\leq h\leq H} \big| Q_{T,h}(s,a) - Q_h^{\star}(s,a) \big| \le\varepsilon
\]
%
with probability exceeding $1-\delta$. 
%
\end{theorem}
%
\begin{remark} 
	\label{remark:expected-error-finite}
	Similar to Remark \ref{remark:expected-error-inf}, an immediate consequence of Theorem~\ref{thm:finite-horizon-simple} is the following mean estimation error bound:
	%
	\[
		\mathbb{E}\Big[ \max_{s,a, h} \big| Q_{T,h}(s,a) - Q_h^{\star}(s,a) \big| \Big] \le 2\varepsilon
	\]
	%
	with the proviso that $T\ge\frac{c_{3}H^4 (\log^{3}T)\big(\log\frac{|\mathcal{S}||\mathcal{A}|H T}{\varepsilon }\big)}{\varepsilon^{2}}$.
\end{remark}
%


For the general scenario (namely, $\{P_h\}_{h=1}^H$ vary across $h$), we collect $|\cS||\cA|H$ independent samples in each iteration, 
and hence Theorem~\ref{thm:finite-horizon-simple} uncovers a sample complexity at most
%
\begin{equation}
	\widetilde{O}\Big( \frac{|\cS||\cA| H^5}{ \varepsilon^2} \Big). 
\end{equation}
%
When it comes to the special case where $P_h\equiv P$ is invariant across all $1\leq h\leq H$, our theorem continues to hold if we generate $|\cS||\cA|$ independent samples per iteration (which include one sample for each state-action pair $(s,a)$ w.r.t.~$P$), leading to a reduced sample complexity of
%
\begin{equation}
	\widetilde{O}\Big( \frac{|\cS||\cA| H^4}{ \varepsilon^2} \Big). 
\end{equation}
%
As it turns out, all this is achieved by means of the same learning rate schedules (i.e., rescaled linear or constant learning rates) as for the discounted infinite-horizon counterpart. 


doubleQ_pieces.tex
Finally, we show that double Q-learning \citep{hasselt2010double} achieves the minimax sample complexity on the constructed hard MDP by mitigating the over-estimation issue. To the best of our knowledge, this is the first case study that provably demonstrates the benefits of double Q-learning over Q-learning from the sample complexity perspective.

Encouragingly, double Q-learning performs over vanilla Q-learning and can still possess minimax optimality in this case, which unveils the efficiency of double Q-learning to some extent.
SyncQlearning.bbl
\begin{thebibliography}{}

\bibitem[Agarwal et~al., 2020]{agarwal2019optimality}
Agarwal, A., Kakade, S., and Yang, L.~F. (2020).
\newblock Model-based reinforcement learning with a generative model is minimax
  optimal.
\newblock {\em Conference on Learning Theory}, pages 67--83.

\bibitem[Azar et~al., 2011]{azar2011reinforcement}
Azar, M.~G., Munos, R., Ghavamzadeh, M., and Kappen, H. (2011).
\newblock Reinforcement learning with a near optimal rate of convergence.
\newblock Technical report, INRIA.

\bibitem[Azar et~al., 2013]{azar2013minimax}
Azar, M.~G., Munos, R., and Kappen, H.~J. (2013).
\newblock Minimax {PAC} bounds on the sample complexity of reinforcement
  learning with a generative model.
\newblock {\em Machine learning}, 91(3):325--349.

\bibitem[Bai et~al., 2019]{bai2019provably}
Bai, Y., Xie, T., Jiang, N., and Wang, Y.-X. (2019).
\newblock Provably efficient $q$-learning with low switching cost.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
  8002--8011.

\bibitem[Beck and Srikant, 2012]{beck2012error}
Beck, C.~L. and Srikant, R. (2012).
\newblock Error bounds for constant step-size {Q}-learning.
\newblock {\em Systems \& control letters}, 61(12):1203--1208.

\bibitem[Bellman, 1952]{bellman1952theory}
Bellman, R. (1952).
\newblock On the theory of dynamic programming.
\newblock {\em Proceedings of the National Academy of Sciences of the United
  States of America}, 38(8):716.

\bibitem[Bertsekas, 2017]{bertsekas2017dynamic}
Bertsekas, D.~P. (2017).
\newblock {\em Dynamic programming and optimal control (4th edition)}.
\newblock Athena Scientific.

\bibitem[Bhandari et~al., 2018]{bhandari2018finite}
Bhandari, J., Russo, D., and Singal, R. (2018).
\newblock A finite time analysis of temporal difference learning with linear
  function approximation.
\newblock In {\em Conference On Learning Theory}, pages 1691--1692.

\bibitem[Bhandari et~al., 2021]{bhandari2021finite}
Bhandari, J., Russo, D., and Singal, R. (2021).
\newblock A finite time analysis of temporal difference learning with linear
  function approximation.
\newblock {\em Operations Research}.

\bibitem[Borkar and Meyn, 2000]{borkar2000ode}
Borkar, V.~S. and Meyn, S.~P. (2000).
\newblock The {ODE} method for convergence of stochastic approximation and
  reinforcement learning.
\newblock {\em SIAM Journal on Control and Optimization}, 38(2):447--469.

\bibitem[Cai et~al., 2019]{cai2019neural}
Cai, Q., Yang, Z., Lee, J.~D., and Wang, Z. (2019).
\newblock Neural temporal-difference and {Q}-learning converges to global
  optima.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
  11312--11322.

\bibitem[Chen et~al., 2020]{chen2020finite}
Chen, Z., Maguluri, S.~T., Shakkottai, S., and Shanmugam, K. (2020).
\newblock Finite-sample analysis of stochastic approximation using smooth
  convex envelopes.
\newblock {\em arXiv preprint arXiv:2002.00874}.

\bibitem[Chen et~al., 2021]{chen2021lyapunov}
Chen, Z., Maguluri, S.~T., Shakkottai, S., and Shanmugam, K. (2021).
\newblock A {L}yapunov theory for finite-sample guarantees of asynchronous
  {Q}-learning and {TD}-learning variants.
\newblock {\em arXiv preprint arXiv:2102.01567}.

\bibitem[Chen et~al., 2019]{chen2019performance}
Chen, Z., Zhang, S., Doan, T.~T., Maguluri, S.~T., and Clarke, J.-P. (2019).
\newblock Performance of {Q}-learning with linear function approximation:
  Stability and finite-time analysis.
\newblock {\em arXiv preprint arXiv:1905.11425}.

\bibitem[Devraj and Meyn, 2020]{devraj2020q}
Devraj, A.~M. and Meyn, S.~P. (2020).
\newblock Q-learning with uniformly bounded variance: Large discounting is not
  a barrier to fast learning.
\newblock {\em arXiv preprint arXiv:2002.10301}.

\bibitem[Doan et~al., 2019]{doan2019finite}
Doan, T., Maguluri, S., and Romberg, J. (2019).
\newblock Finite-time analysis of distributed {TD}(0) with linear function
  approximation on multi-agent reinforcement learning.
\newblock In {\em International Conference on Machine Learning}, pages
  1626--1635. PMLR.

\bibitem[Even-Dar and Mansour, 2003]{even2003learning}
Even-Dar, E. and Mansour, Y. (2003).
\newblock Learning rates for {Q}-learning.
\newblock {\em Journal of machine learning Research}, 5(Dec):1--25.

\bibitem[Fan et~al., 2019]{yang2019theoretical}
Fan, J., Wang, Z., Xie, Y., and Yang, Z. (2019).
\newblock A theoretical analysis of deep {Q}-learning.
\newblock {\em arXiv preprint arXiv:1901.00137}.

\bibitem[Freedman, 1975]{freedman1975tail}
Freedman, D.~A. (1975).
\newblock On tail probabilities for martingales.
\newblock {\em the Annals of Probability}, pages 100--118.

\bibitem[Gupta et~al., 2019]{gupta2019finite}
Gupta, H., Srikant, R., and Ying, L. (2019).
\newblock Finite-time performance bounds and adaptive learning rate selection
  for two time-scale reinforcement learning.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
  4706--4715.

\bibitem[Hasselt, 2010]{hasselt2010double}
Hasselt, H. (2010).
\newblock Double {Q}-learning.
\newblock {\em Advances in neural information processing systems},
  23:2613--2621.

\bibitem[Jaakkola et~al., 1994]{jaakkola1994convergence}
Jaakkola, T., Jordan, M.~I., and Singh, S.~P. (1994).
\newblock Convergence of stochastic iterative dynamic programming algorithms.
\newblock In {\em Advances in neural information processing systems}, pages
  703--710.

\bibitem[Jin et~al., 2018]{jin2018q}
Jin, C., Allen-Zhu, Z., Bubeck, S., and Jordan, M.~I. (2018).
\newblock Is {Q}-learning provably efficient?
\newblock In {\em Advances in Neural Information Processing Systems}, pages
  4863--4873.

\bibitem[Kakade, 2003]{kakade2003sample}
Kakade, S. (2003).
\newblock {\em On the sample complexity of reinforcement learning}.
\newblock PhD thesis, University of London.

\bibitem[Kearns et~al., 2002]{kearns2002sparse}
Kearns, M., Mansour, Y., and Ng, A.~Y. (2002).
\newblock A sparse sampling algorithm for near-optimal planning in large
  {M}arkov decision processes.
\newblock {\em Machine learning}, 49(2-3):193--208.

\bibitem[Kearns and Singh, 1999]{kearns1999finite}
Kearns, M.~J. and Singh, S.~P. (1999).
\newblock Finite-sample convergence rates for {Q}-learning and indirect
  algorithms.
\newblock In {\em Advances in neural information processing systems}, pages
  996--1002.

\bibitem[Khamaru et~al., 2020]{khamaru2020temporal}
Khamaru, K., Pananjady, A., Ruan, F., Wainwright, M.~J., and Jordan, M.~I.
  (2020).
\newblock Is temporal difference learning optimal? an instance-dependent
  analysis.
\newblock {\em arXiv preprint arXiv:2003.07337}.

\bibitem[Khamaru et~al., 2021]{khamaru2021temporal}
Khamaru, K., Pananjady, A., Ruan, F., Wainwright, M.~J., and Jordan, M.~I.
  (2021).
\newblock Is temporal difference learning optimal? an instance-dependent
  analysis.
\newblock {\em SIAM Journal on Mathematics of Data Science}, 3(4):1013--1040.

\bibitem[Lakshminarayanan and Szepesvari, 2018]{lakshminarayanan2018linear}
Lakshminarayanan, C. and Szepesvari, C. (2018).
\newblock Linear stochastic approximation: How far does constant step-size and
  iterate averaging go?
\newblock In {\em International Conference on Artificial Intelligence and
  Statistics}, pages 1347--1355.

\bibitem[Lee and He, 2018]{lee2018stochastic}
Lee, D. and He, N. (2018).
\newblock Stochastic primal-dual {Q}-learning.
\newblock {\em arXiv preprint arXiv:1810.08298}.

\bibitem[Li et~al., 2021a]{li2021breaking}
Li, G., Shi, L., Chen, Y., Gu, Y., and Chi, Y. (2021a).
\newblock Breaking the sample complexity barrier to regret-optimal model-free
  reinforcement learning.
\newblock {\em Neural Information Processing Systems (NeurIPS)}.

\bibitem[Li et~al., 2020]{li2020breaking}
Li, G., Wei, Y., Chi, Y., Gu, Y., and Chen, Y. (2020).
\newblock Breaking the sample size barrier in model-based reinforcement
  learning with a generative model.
\newblock {\em Advances in Neural Information Processing Systems}, 33.

\bibitem[Li et~al., 2021b]{li2020sample}
Li, G., Wei, Y., Chi, Y., Gu, Y., and Chen, Y. (2021b).
\newblock Sample complexity of asynchronous {Q}-learning: Sharper analysis and
  variance reduction.
\newblock {\em IEEE Transactions on Information Theory}.

\bibitem[Mou et~al., 2020]{mou2020linear}
Mou, W., Li, C.~J., Wainwright, M.~J., Bartlett, P.~L., and Jordan, M.~I.
  (2020).
\newblock On linear stochastic approximation: Fine-grained {P}olyak-{R}uppert
  and non-asymptotic concentration.
\newblock {\em arXiv preprint arXiv:2004.04719}.

\bibitem[Murphy, 2005]{murphy2005generalization}
Murphy, S. (2005).
\newblock A generalization error for {Q}-learning.
\newblock {\em Journal of Machine Learning Research}, 6:1073--1097.

\bibitem[Pananjady and Wainwright, 2020]{pananjady2019value}
Pananjady, A. and Wainwright, M.~J. (2020).
\newblock Instance-dependent $\ell_{\infty}$-bounds for policy evaluation in
  tabular reinforcement learning.
\newblock {\em IEEE Transactions on Information Theory}, 67(1):566--585.

\bibitem[Paulin, 2015]{paulin2015concentration}
Paulin, D. (2015).
\newblock Concentration inequalities for {M}arkov chains by {M}arton couplings
  and spectral methods.
\newblock {\em Electronic Journal of Probability}, 20.

\bibitem[Qu and Wierman, 2020]{qu2020finite}
Qu, G. and Wierman, A. (2020).
\newblock Finite-time analysis of asynchronous stochastic approximation and
  {Q}-learning.
\newblock {\em Conference on Learning Theory}, pages 3185--3205.

\bibitem[Robbins and Monro, 1951]{robbins1951stochastic}
Robbins, H. and Monro, S. (1951).
\newblock A stochastic approximation method.
\newblock {\em The annals of mathematical statistics}, pages 400--407.

\bibitem[Shah and Xie, 2018]{shah2018q}
Shah, D. and Xie, Q. (2018).
\newblock Q-learning with nearest neighbors.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
  3111--3121.

\bibitem[Sidford et~al., 2018]{sidford2018near}
Sidford, A., Wang, M., Wu, X., Yang, L., and Ye, Y. (2018).
\newblock Near-optimal time and sample complexities for solving {M}arkov
  decision processes with a generative model.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
  5186--5196.

\bibitem[Srikant and Ying, 2019]{srikant2019finite}
Srikant, R. and Ying, L. (2019).
\newblock Finite-time error bounds for linear stochastic approximation and {TD}
  learning.
\newblock In {\em Conference on Learning Theory}, pages 2803--2830.

\bibitem[Sutton, 1988]{sutton1988learning}
Sutton, R.~S. (1988).
\newblock Learning to predict by the methods of temporal differences.
\newblock {\em Machine learning}, 3(1):9--44.

\bibitem[Sutton and Barto, 2018]{sutton2018reinforcement}
Sutton, R.~S. and Barto, A.~G. (2018).
\newblock {\em Reinforcement learning: An introduction}.
\newblock MIT press.

\bibitem[Szepesv{\'a}ri, 1998]{szepesvari1998asymptotic}
Szepesv{\'a}ri, C. (1998).
\newblock The asymptotic convergence-rate of {Q}-learning.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
  1064--1070.

\bibitem[Tropp, 2011]{tropp2011freedman}
Tropp, J. (2011).
\newblock Freedman's inequality for matrix martingales.
\newblock {\em Electronic Communications in Probability}, 16:262--270.

\bibitem[Tsitsiklis and Van~Roy, 1997]{tsitsiklis1997analysis}
Tsitsiklis, J. and Van~Roy, B. (1997).
\newblock An analysis of temporal-difference learning with function
  approximation.
\newblock {\em IEEE Transactions on Automatic Control}, 42(5):674--690.

\bibitem[Tsitsiklis, 1994]{tsitsiklis1994asynchronous}
Tsitsiklis, J.~N. (1994).
\newblock Asynchronous stochastic approximation and {Q}-learning.
\newblock {\em Machine learning}, 16(3):185--202.

\bibitem[Tsybakov and Zaiats, 2009]{tsybakov2009introduction}
Tsybakov, A.~B. and Zaiats, V. (2009).
\newblock {\em Introduction to nonparametric estimation}, volume~11.
\newblock Springer.

\bibitem[Wai et~al., 2019]{wai2019variance}
Wai, H.-T., Hong, M., Yang, Z., Wang, Z., and Tang, K. (2019).
\newblock Variance reduced policy evaluation with smooth function
  approximation.
\newblock {\em Advances in Neural Information Processing Systems},
  32:5784--5795.

\bibitem[Wainwright, 2019a]{wainwright2019high}
Wainwright, M. (2019a).
\newblock {\em High-Dimensional Statistics: A Non-Asymptotic Viewpoint}.
\newblock Cambridge Series in Statistical and Probabilistic Mathematics.
  Cambridge University Press.

\bibitem[Wainwright, 2019b]{wainwright2019stochastic}
Wainwright, M.~J. (2019b).
\newblock Stochastic approximation with cone-contractive operators: Sharp
  $\ell_{\infty}$-bounds for {Q}-learning.
\newblock {\em arXiv preprint arXiv:1905.06265}.

\bibitem[Wainwright, 2019c]{wainwright2019variance}
Wainwright, M.~J. (2019c).
\newblock Variance-reduced {Q}-learning is minimax optimal.
\newblock {\em arXiv preprint arXiv:1906.04697}.

\bibitem[Watkins and Dayan, 1992]{watkins1992q}
Watkins, C.~J. and Dayan, P. (1992).
\newblock {Q}-learning.
\newblock {\em Machine learning}, 8(3-4):279--292.

\bibitem[Watkins, 1989]{watkins1989learning}
Watkins, C. J. C.~H. (1989).
\newblock Learning from delayed rewards.

\bibitem[Weng et~al., 2020a]{weng2020momentum}
Weng, B., Xiong, H., Zhao, L., Liang, Y., and Zhang, W. (2020a).
\newblock Momentum {Q}-learning with finite-sample convergence guarantee.
\newblock {\em arXiv preprint arXiv:2007.15418}.

\bibitem[Weng et~al., 2020b]{weng2020mean}
Weng, W., Gupta, H., He, N., Ying, L., and Srikant, R. (2020b).
\newblock The mean-squared error of double {Q}-learning.
\newblock {\em Advances in Neural Information Processing Systems}, 33.

\bibitem[Wu et~al., 2020]{wu2020finite}
Wu, Y., Zhang, W., Xu, P., and Gu, Q. (2020).
\newblock A finite time analysis of two time-scale actor critic methods.
\newblock {\em arXiv preprint arXiv:2005.01350}.

\bibitem[Xiong et~al., 2020]{xiong2020finitetime}
Xiong, H., Zhao, L., Liang, Y., and Zhang, W. (2020).
\newblock Finite-time analysis for double {Q}-learning.
\newblock {\em Advances in Neural Information Processing Systems}, 33.

\bibitem[Xu and Gu, 2020]{xu2020finite}
Xu, P. and Gu, Q. (2020).
\newblock A finite-time analysis of {Q}-learning with neural network function
  approximation.
\newblock In {\em International Conference on Machine Learning}, pages
  10555--10565. PMLR.

\bibitem[Xu et~al., 2019a]{xu2019reanalysis}
Xu, T., Wang, Z., Zhou, Y., and Liang, Y. (2019a).
\newblock Reanalysis of variance reduced temporal difference learning.
\newblock In {\em International Conference on Learning Representations}.

\bibitem[Xu et~al., 2019b]{xu2019two}
Xu, T., Zou, S., and Liang, Y. (2019b).
\newblock Two time-scale off-policy {TD} learning: Non-asymptotic analysis over
  {M}arkovian samples.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
  10633--10643.

\bibitem[Zhang et~al., 2020]{zhang2020almost}
Zhang, Z., Zhou, Y., and Ji, X. (2020).
\newblock Almost optimal model-free reinforcement learningvia
  reference-advantage decomposition.
\newblock {\em Advances in Neural Information Processing Systems}, 33.

\end{thebibliography}

bibfileRL.bib
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%  citations from MR lookup

@misc{NCDC,
  title = {National climatic data center},
  howpublished = {\url{https://www.ncdc.noaa.gov/}},
  note = {Accessed: 2019-08-31},
  year={2019}
}

@book{sutton2018reinforcement,
  title={Reinforcement learning: An introduction},
  author={Sutton, Richard S and Barto, Andrew G},
  year={2018},
  publisher={MIT press}
}

 @article{hasselt2010double,
  title={Double {Q}-learning},
  author={Hasselt, Hado},
  journal={Advances in neural information processing systems},
  volume={23},
  pages={2613--2621},
  year={2010},
  publisher={Citeseer}
}

@article{li2020breaking,
  title={Breaking the sample size barrier in model-based reinforcement learning with a generative model},
  author={Li, Gen and Wei, Yuting and Chi, Yuejie and Gu, Yuantao and Chen, Yuxin},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}

@article{murphy2005generalization,
  title={A Generalization Error for {Q}-Learning},
  author={Murphy, Susan},
  journal={Journal of Machine Learning Research},
  volume={6},
  pages={1073--1097},
  year={2005}
}

@article{robbins1951stochastic,
  title={A stochastic approximation method},
  author={Robbins, Herbert and Monro, Sutton},
  journal={The annals of mathematical statistics},
  pages={400--407},
  year={1951},
  publisher={JSTOR}
}

@article{xiong2020finitetime,
  title={Finite-time analysis for double {Q}-learning},
  author={Xiong, Huaqing and Zhao, Lin and Liang, Yingbin and Zhang, Wei},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
} 


@article{chen2021lyapunov,
  title={A {L}yapunov Theory for Finite-Sample Guarantees of Asynchronous {Q}-Learning and {TD}-Learning Variants},
  author={Chen, Zaiwei and Maguluri, Siva Theja and Sanjay Shakkottai and Karthikeyan Shanmugam},
  journal={arXiv preprint arXiv:2102.01567},
  year={2021}
}



@article{tropp2011freedman,
  title={Freedman's inequality for matrix martingales},
  author={Tropp, Joel},
  journal={Electronic Communications in Probability},
  volume={16},
  pages={262--270},
  year={2011},
  publisher={The Institute of Mathematical Statistics and the Bernoulli Society}
}


@article{sutton1988learning,
  title={Learning to predict by the methods of temporal differences},
  author={Sutton, Richard S},
  journal={Machine learning},
  volume={3},
  number={1},
  pages={9--44},
  year={1988},
  publisher={Springer}
}

@article{tropp2011freedman,
  title={Freedman's inequality for matrix martingales},
  author={Tropp, Joel},
  journal={Electronic Communications in Probability},
  volume={16},
  pages={262--270},
  year={2011},
  publisher={The Institute of Mathematical Statistics and the Bernoulli Society}
}

@article{freedman1975tail,
  title={On tail probabilities for martingales},
  author={Freedman, David A},
  journal={the Annals of Probability},
  pages={100--118},
  year={1975},
  publisher={JSTOR}
}

@article{weng2020momentum,
  title={Momentum {Q}-learning with Finite-Sample Convergence Guarantee},
  author={Weng, Bowen and Xiong, Huaqing and Zhao, Lin and Liang, Yingbin and Zhang, Wei},
  journal={arXiv preprint arXiv:2007.15418},
  year={2020}
}


@inproceedings{yang2019sample,
  title={Sample-optimal parametric {Q}-learning using linearly additive features},
  author={Yang, Lin and Wang, Mengdi},
  booktitle={International Conference on Machine Learning},
  pages={6995--7004},
  year={2019}
}

@article{mnih2015human,
  title={Human-level control through deep reinforcement learning},
  author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Rusu, Andrei A and Veness, Joel and Bellemare, Marc G and Graves, Alex and Riedmiller, Martin and Fidjeland, Andreas K and Ostrovski, Georg},
  journal={Nature},
  volume={518},
  number={7540},
  pages={529--533},
  year={2015},
  publisher={Nature Publishing Group}
}

@article{lin2020distributed,
  title={Distributed Reinforcement Learning in Multi-Agent Networked Systems},
  author={Lin, Yiheng and Qu, Guannan and Huang, Longbo and Wierman, Adam},
  journal={arXiv preprint arXiv:2006.06555},
  year={2020}
}


@inproceedings{ghavamzadeh2011speedy,
  title={Speedy {Q}-learning},
  author={Azar, Mohammad G and Kappen, Hilbert J and Ghavamzadeh, Mohammad and   Munos, R{\'e}mi},
  booktitle={Advances in neural information processing systems},
  pages={2411--2419},
  year={2011}
}

@inproceedings{zou2019finite,
  title={Finite-sample analysis for {SARSA} with linear function approximation},
  author={Zou, Shaofeng and Xu, Tengyu and Liang, Yingbin},
  booktitle={Advances in Neural Information Processing Systems},
  pages={8665--8675},
  year={2019}
}

@book{bremaud2013markov,
  title={Markov chains: {G}ibbs fields, {M}onte Carlo simulation, and queues},
  author={Br{\'e}maud, Pierre},
  volume={31},
  year={2013},
  publisher={Springer Science \& Business Media}
}

@article{paulin2015concentration,
  title={Concentration inequalities for {M}arkov chains by {M}arton couplings and spectral methods},
  author={Paulin, Daniel},
  journal={Electronic Journal of Probability},
  volume={20},
  year={2015},
  publisher={The Institute of Mathematical Statistics and the Bernoulli Society}
}

@article{jiang2018bernstein,
  title={Bernstein's inequality for general {M}arkov chains},
  author={Jiang, Bai and Sun, Qiang and Fan, Jianqing},
  journal={arXiv preprint arXiv:1805.10721},
  year={2018}
}

@article{li2020breaking,
    title={Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model},
    author={Gen Li and Yuting Wei and Yuejie Chi and Yuantao Gu and Yuxin Chen},
    year={2020},
     journal={arXiv preprint arXiv:2005.12900, accepted to Neural Information Processing Systems}
}

@inproceedings{du2019provably,
  title={Provably efficient {Q}-learning with function approximation via distribution shift error checking oracle},
  author={Du, Simon S and Luo, Yuping and Wang, Ruosong and Zhang, Hanrui},
  booktitle={Advances in Neural Information Processing Systems},
  pages={8058--8068},
  year={2019}
}

@techreport{azar2011reinforcement,
  title={Reinforcement learning with a near optimal rate of convergence},
  author={Azar, Mohammad Gheshlaghi and Munos, R{\'e}mi and Ghavamzadeh, Mohammad and Kappen, Hilbert},
  institution={INRIA},
  year={2011}
}

@article{kaledin2020finite,
  title={Finite Time Analysis of Linear Two-timescale Stochastic Approximation with {M}arkovian Noise},
  author={Kaledin, Maxim and Moulines, Eric and Naumov, Alexey and Tadic, Vladislav and Wai, Hoi-To},
  journal={arXiv preprint arXiv:2002.01268},
  year={2020}
}

@inproceedings{dalal2018finite_two,
  title={Finite Sample Analysis of Two-Timescale Stochastic Approximation with Applications to Reinforcement Learning},
  author={Dalal, Gal and Thoppe, Gugan and Sz{\"o}r{\'e}nyi, Bal{\'a}zs and Mannor, Shie},
  booktitle={Conference On Learning Theory},
  pages={1199--1233},
  year={2018}
}

@inproceedings{dalal2018finite,
  title={Finite sample analyses for {TD}(0) with function approximation},
  author={Dalal, Gal and Sz{\"o}r{\'e}nyi, Bal{\'a}zs and Thoppe, Gugan and Mannor, Shie},
  booktitle={Thirty-Second AAAI Conference on Artificial Intelligence},
  year={2018}
}

@inproceedings{Wang2020Qlearning,
title={{Q}-learning with {UCB} Exploration is Sample Efficient for Infinite-Horizon {MDP}},
author={Yuanhao Wang and Kefan Dong and Xiaoyu Chen and Liwei Wang},
booktitle={International Conference on Learning Representations},
year={2020}
}

@inproceedings{du2017stochastic,
  title={Stochastic variance reduction methods for policy evaluation},
  author={Du, Simon S and Chen, Jianshu and Li, Lihong and Xiao, Lin and Zhou, Dengyong},
  booktitle={Proceedings of the 34th International Conference on Machine Learning-Volume 70},
  pages={1049--1058},
  year={2017},
  organization={JMLR. org}
}


@article{chen2019performance,
  title={Performance of {Q}-learning with linear function approximation: Stability and finite-time analysis},
  author={Chen, Zaiwei and Zhang, Sheng and Doan, Thinh T and Maguluri, Siva Theja and Clarke, John-Paul},
  journal={arXiv preprint arXiv:1905.11425},
  year={2019}
}

@article{yang2019theoretical,
  title={A theoretical analysis of deep {Q}-learning},
  author={Jianqing Fan and Zhaoran Wang and Xie, Yuchen and Yang, Zhuoran},
  journal={arXiv preprint arXiv:1901.00137},
  year={2019}
}

@article{devraj2020q,
  title={Q-learning with uniformly bounded variance: Large discounting is not a barrier to fast learning},
  author={Devraj, Adithya M and Meyn, Sean P},
  journal={arXiv preprint arXiv:2002.10301},
  year={2020}
}

@article{chen2020finite,
  title={Finite-Sample Analysis of Stochastic Approximation Using Smooth Convex Envelopes},
  author={Chen, Zaiwei and Maguluri, Siva Theja and Shakkottai, Sanjay and Shanmugam, Karthikeyan},
  journal={arXiv preprint arXiv:2002.00874},
  year={2020}
}

@article{wai2019variance,
  title={Variance reduced policy evaluation with smooth function approximation},
  author={Wai, Hoi-To and Hong, Mingyi and Yang, Zhuoran and Wang, Zhaoran and Tang, Kexin},
  journal={Advances in Neural Information Processing Systems},
  volume={32},
  pages={5784--5795},
  year={2019}
}

@article{chen2019performance,
  title={Performance of {Q}-learning with linear function approximation: Stability and finite-time analysis},
  author={Chen, Zaiwei and Zhang, Sheng and Doan, Thinh T and Maguluri, Siva Theja and Clarke, John-Paul},
  journal={arXiv preprint arXiv:1905.11425},
  year={2019}
}

@inproceedings{doan2019finite,
  title={Finite-time analysis of distributed {TD}(0) with linear function approximation on multi-agent reinforcement learning},
  author={Doan, Thinh and Maguluri, Siva and Romberg, Justin},
  booktitle={International Conference on Machine Learning},
  pages={1626--1635},
  year={2019},
  organization={PMLR}
}

@inproceedings{szepesvari1998asymptotic,
  title={The asymptotic convergence-rate of {Q}-learning},
  author={Szepesv{\'a}ri, Csaba},
  booktitle={Advances in Neural Information Processing Systems},
  pages={1064--1070},
  year={1998}
}

@article{beck2012error,
  title={Error bounds for constant step-size {Q}-learning},
  author={Beck, Carolyn L and Srikant, Rayadurgam},
  journal={Systems \& control letters},
  volume={61},
  number={12},
  pages={1203--1208},
  year={2012},
  publisher={Elsevier}
}

@article{azar2012sample,
  title={On the sample complexity of reinforcement learning with a generative model},
  author={Azar, Mohammad Gheshlaghi and Munos, R{\'e}mi and Kappen, Bert},
  journal={arXiv preprint arXiv:1206.6461},
  year={2012}
}

@inproceedings{jaakkola1994convergence,
  title={Convergence of stochastic iterative dynamic programming algorithms},
  author={Jaakkola, Tommi and Jordan, Michael I and Singh, Satinder P},
  booktitle={Advances in neural information processing systems},
  pages={703--710},
  year={1994}
}

@article{even2003learning,
  title={Learning rates for {Q}-learning},
  author={Even-Dar, Eyal and Mansour, Yishay},
  journal={Journal of machine learning Research},
  volume={5},
  number={Dec},
  pages={1--25},
  year={2003}
}

@article{watkins1992q,
  title={{Q}-learning},
  author={Watkins, Christopher JCH and Dayan, Peter},
  journal={Machine learning},
  volume={8},
  number={3-4},
  pages={279--292},
  year={1992},
  publisher={Springer}
}

@inproceedings{xu2020finite,
  title={A finite-time analysis of {Q}-learning with neural network function approximation},
  author={Xu, Pan and Gu, Quanquan},
  booktitle={International Conference on Machine Learning},
  pages={10555--10565},
  year={2020},
  organization={PMLR}
}

@inproceedings{xu2019reanalysis,
  title={Reanalysis of Variance Reduced Temporal Difference Learning},
  author={Xu, Tengyu and Wang, Zhe and Zhou, Yi and Liang, Yingbin},
  booktitle={International Conference on Learning Representations},
  year={2019}
}

@article{lee2018stochastic,
  title={Stochastic primal-dual {Q}-learning},
  author={Lee, Donghwan and He, Niao},
  journal={arXiv preprint arXiv:1810.08298},
  year={2018}
}

@article{weng2020mean,
  title={The Mean-Squared Error of Double {Q}-Learning},
  author={Weng, Wentao and Gupta, Harsh and He, Niao and Ying, Lei and Srikant, R},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}

@article{wu2020finite,
  title={A finite time analysis of two time-scale actor critic methods},
  author={Wu, Yue and Zhang, Weitong and Xu, Pan and Gu, Quanquan},
  journal={arXiv preprint arXiv:2005.01350},
  year={2020}
}

@inproceedings{gupta2019finite,
  title={Finite-time performance bounds and adaptive learning rate selection for two time-scale reinforcement learning},
  author={Gupta, Harsh and Srikant, R and Ying, Lei},
  booktitle={Advances in Neural Information Processing Systems},
  pages={4706--4715},
  year={2019}
}

@inproceedings{szepesvari1998asymptotic,
  title={The asymptotic convergence-rate of {Q}-learning},
  author={Szepesv{\'a}ri, Csaba},
  booktitle={Advances in Neural Information Processing Systems},
  pages={1064--1070},
  year={1998}
}

@inproceedings{xu2019two,
  title={Two time-scale off-policy {TD} learning: Non-asymptotic analysis over {M}arkovian samples},
  author={Xu, Tengyu and Zou, Shaofeng and Liang, Yingbin},
  booktitle={Advances in Neural Information Processing Systems},
  pages={10633--10643},
  year={2019}
}

@article{bellman1952theory,
  title={On the theory of dynamic programming},
  author={Bellman, Richard},
  journal={Proceedings of the National Academy of Sciences of the United States of America},
  volume={38},
  number={8},
  pages={716},
  year={1952},
  publisher={National Academy of Sciences}
}

@inproceedings{lattimore2012pac,
  title={{PAC bounds for discounted MDPs}},
  author={Lattimore, Tor and Hutter, Marcus},
  booktitle={International Conference on Algorithmic Learning Theory},
  pages={320--334},
  year={2012},
  organization={Springer}
}

@article{lazaric2012finite,
  title={Finite-sample analysis of least-squares policy iteration},
  author={Lazaric, Alessandro and Ghavamzadeh, Mohammad and Munos, R{\'e}mi},
  journal={Journal of Machine Learning Research},
  volume={13},
  number={Oct},
  pages={3041--3074},
  year={2012}
}

@article{bradtke1996linear,
  title={Linear least-squares algorithms for temporal difference learning},
  author={Bradtke, Steven J and Barto, Andrew G},
  journal={Machine learning},
  volume={22},
  number={1-3},
  pages={33--57},
  year={1996},
  publisher={Springer}
}

@inproceedings{jin2018q,
  title={Is {Q}-learning provably efficient?},
  author={Jin, Chi and Allen-Zhu, Zeyuan and Bubeck, Sebastien and Jordan, Michael I},
  booktitle={Advances in Neural Information Processing Systems},
  pages={4863--4873},
  year={2018}
}

@inproceedings{lakshminarayanan2018linear,
  title={Linear stochastic approximation: How far does constant step-size and iterate averaging go?},
  author={Lakshminarayanan, Chandrashekar and Szepesvari, Csaba},
  booktitle={International Conference on Artificial Intelligence and Statistics},
  pages={1347--1355},
  year={2018}
}

@article{szepesvari2010algorithms,
  title={Algorithms for reinforcement learning},
  author={Szepesv{\'a}ri, Csaba},
  journal={Synthesis lectures on artificial intelligence and machine learning},
  volume={4},
  number={1},
  pages={1--103},
  year={2010},
  publisher={Morgan \& Claypool Publishers}
}

@book{bertsekas2017dynamic,
  title={Dynamic programming and optimal control (4th edition)},
  author={Bertsekas, Dimitri P},
  year = {2017},
  publisher={Athena Scientific}
}

@article{sun2020markov,
  title={Markov chain block coordinate descent},
  author={Sun, Tao and Sun, Yuejiao and Xu, Yangyang and Yin, Wotao},
  journal={Computational Optimization and Applications},
  pages={1--27},
  year={2020},
  publisher={Springer}
}


@article{qu2020finite,
  title={Finite-Time Analysis of Asynchronous Stochastic Approximation and {Q}-Learning},
  author={Qu, Guannan and Wierman, Adam},
  journal={Conference on Learning Theory},
  pages={3185--3205},
  year={2020},
}

@inproceedings{doan2019finite,
  title={Finite-time analysis of distributed {TD}(0) with linear function approximation on multi-agent reinforcement learning},
  author={Doan, Thinh and Maguluri, Siva and Romberg, Justin},
  booktitle={International Conference on Machine Learning},
  pages={1626--1635},
  year={2019}
}

@article{khamaru2020temporal,
  title={Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis},
  author={Khamaru, Koulik and Pananjady, Ashwin and Ruan, Feng and Wainwright, Martin J and Jordan, Michael I},
  journal={arXiv preprint arXiv:2003.07337},
  year={2020}
}

@inproceedings{shah2018q,
  title={Q-learning with nearest neighbors},
  author={Shah, Devavrat and Xie, Qiaomin},
  booktitle={Advances in Neural Information Processing Systems},
  pages={3111--3121},
  year={2018}
}

@inproceedings{bai2019provably,
  title={Provably efficient $Q$-learning with low switching cost},
  author={Bai, Yu and Xie, Tengyang and Jiang, Nan and Wang, Yu-Xiang},
  booktitle={Advances in Neural Information Processing Systems},
  pages={8002--8011},
  year={2019}
}

@inproceedings{strehl2006pac,
  title={{PAC} model-free reinforcement learning},
  author={Strehl, Alexander L and Li, Lihong and Wiewiora, Eric and Langford, John and Littman, Michael L},
  booktitle={Proceedings of the 23rd international conference on Machine learning},
  pages={881--888},
  year={2006}
}

@inproceedings{jiang2018open,
  title={Open problem: The dependence of sample complexity lower bounds on planning horizon},
  author={Jiang, Nan and Agarwal, Alekh},
  booktitle={Conference On Learning Theory},
  pages={3395--3398},
  year={2018}
}

@inproceedings{dann2015sample,
  title={Sample complexity of episodic fixed-horizon reinforcement learning},
  author={Dann, Christoph and Brunskill, Emma},
  booktitle={Advances in Neural Information Processing Systems},
  pages={2818--2826},
  year={2015}
}

@article{jin2019provably,
  title={Provably efficient reinforcement learning with linear function approximation},
  author={Jin, Chi and Yang, Zhuoran and Wang, Zhaoran and Jordan, Michael I},
  journal={arXiv preprint arXiv:1907.05388},
  year={2019}
}

@article{wainwright2019variance,
  title={Variance-reduced {Q}-learning is minimax optimal},
  author={Wainwright, Martin J},
  journal={arXiv preprint arXiv:1906.04697},
  year={2019}
}

@inproceedings{azar2017minimax,
  title={Minimax regret bounds for reinforcement learning},
  author={Azar, Mohammad Gheshlaghi and Osband, Ian and Munos, R{\'e}mi},
  booktitle={Proceedings of the 34th International Conference on Machine Learning-Volume 70},
  pages={263--272},
  year={2017},
  organization={JMLR. org}
}

@article{lee2019target,
  title={Target-based temporal difference learning},
  author={Lee, Donghwan and He, Niao},
  journal={arXiv preprint arXiv:1904.10945},
  year={2019}
} 

@article{bai2019provably,
  title={Provably efficient {Q}-learning with low switching cost},
  author={Bai, Yu and Xie, Tengyang and Jiang, Nan and Wang, Yu Xiang},
  journal={Advances in Neural Information Processing Systems},
  volume={32},
  year={2019}
}

@article{zhang2020almost,
  title={Almost Optimal Model-Free Reinforcement Learningvia Reference-Advantage Decomposition},
  author={Zhang, Zihan and Zhou, Yuan and Ji, Xiangyang},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}

@article{doan2020convergence,
  title={Convergence rates of accelerated markov gradient descent with applications in reinforcement learning},
  author={Doan, Thinh T and Nguyen, Lam M and Pham, Nhan H and Romberg, Justin},
  journal={arXiv preprint arXiv:2002.02873},
  year={2020}
}

@article{li2021breaking,
  title={Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learning},
  author={Li, Gen and Shi, Laixi and Chen, Yuxin and Gu, Yuantao and Chi, Yuejie},
  journal={Neural Information Processing Systems (NeurIPS)},
  year={2021}
}

@article{sutton1988learning,
  title={Learning to predict by the methods of temporal differences},
  author={Sutton, Richard S},
  journal={Machine learning},
  volume={3},
  number={1},
  pages={9--44},
  year={1988},
  publisher={Springer}
}

@inproceedings{srikant2019finite,
  title={Finite-Time Error Bounds For Linear Stochastic Approximation and {TD} Learning},
  author={Srikant, R and Ying, Lei},
  booktitle={Conference on Learning Theory},
  pages={2803--2830},
  year={2019}
}

@article{chen2019spectral,
  title={Spectral method and regularized MLE are both optimal for top-K ranking},
  author={Chen, Yuxin and Fan, Jianqing and Ma, Cong and Wang, Kaizheng},
  journal={Annals of statistics},
  volume={47},
  number={4},
  pages={2204},
  year={2019},
  publisher={NIH Public Access}
}

@article{tsitsiklis1997analysis,
  title={An Analysis of Temporal-Difference Learning with Function Approximation},
  author={Tsitsiklis, John and Van Roy, Benjamin},
  journal={IEEE Transactions on Automatic Control},
  volume={42},
  number={5},
  pages={674--690},
  year={1997},
}

@article{borkar2000ode,
  title={The {ODE} method for convergence of stochastic approximation and reinforcement learning},
  author={Borkar, Vivek S and Meyn, Sean P},
  journal={SIAM Journal on Control and Optimization},
  volume={38},
  number={2},
  pages={447--469},
  year={2000},
  publisher={SIAM}
}


@article{bhandari2021finite,
  title={A Finite Time Analysis of Temporal Difference Learning with Linear Function Approximation},
  author={Bhandari, Jalaj and Russo, Daniel and Singal, Raghav},
  journal={Operations Research},
  year={2021},
  publisher={INFORMS}
}

@inproceedings{bhandari2018finite,
  title={A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation},
  author={Bhandari, Jalaj and Russo, Daniel and Singal, Raghav},
  booktitle={Conference On Learning Theory},
  pages={1691--1692},
  year={2018}
}

 

@article{li2020sample,
  title={Sample Complexity of Asynchronous {Q}-Learning: Sharper Analysis and Variance Reduction},
  author={Li, Gen and Wei, Yuting and Chi, Yuejie and Gu, Yuantao and Chen, Yuxin},
  journal={IEEE Transactions on Information Theory},
  year={2021},
  publisher={IEEE}
}

@inproceedings{shah2018q,
  title={Q-learning with nearest neighbors},
  author={Shah, Devavrat and Xie, Qiaomin},
  booktitle={Proceedings of the 32nd International Conference on Neural Information Processing Systems},
  pages={3115--3125},
  year={2018}
}


@article{mou2020linear,
  title={On Linear Stochastic Approximation: Fine-grained {P}olyak-{R}uppert and Non-Asymptotic Concentration},
  author={Mou, Wenlong and Li, Chris Junchi and Wainwright, Martin J and Bartlett, Peter L and Jordan, Michael I},
  journal={arXiv preprint arXiv:2004.04719},
  year={2020}
}

@article{tsitsiklis1994asynchronous,
  title={Asynchronous stochastic approximation and {Q}-learning},
  author={Tsitsiklis, John N},
  journal={Machine learning},
  volume={16},
  number={3},
  pages={185--202},
  year={1994},
  publisher={Springer}
}

@inproceedings{cai2019neural,
  title={Neural temporal-difference and {Q}-Learning converges to global optima},
  author={Cai, Qi and Yang, Zhuoran and Lee, Jason D and Wang, Zhaoran},
  booktitle={Advances in Neural Information Processing Systems},
  pages={11312--11322},
  year={2019}
}

@article{chen2016stochastic,
  title={Stochastic primal-dual methods and sample complexity of reinforcement learning},
  author={Chen, Yichen and Wang, Mengdi},
  journal={arXiv preprint arXiv:1612.02516},
  year={2016}
}

@article{xu2019finite,
  title={A Finite-Time Analysis of {Q}-Learning with Neural Network Function Approximation},
  author={Xu, Pan and Gu, Quanquan},
  journal={accepted to International Conference on Machine Learning},
  year={2020}
}

@article{kakade2018variance,
  title={Variance reduction methods for sublinear reinforcement learning},
  author={Kakade, Sham and Wang, Mengdi and Yang, Lin F},
  journal={arXiv preprint arXiv:1802.09184},
  year={2018}
} 

@article{duan2020minimax,
  title={Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation},
  author={Duan, Yaqi and Wang, Mengdi},
  journal={arXiv preprint arXiv:2002.09516},
  year={2020}
}


@article{khamaru2020temporal,
  title={Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis},
  author={Khamaru, Koulik and Pananjady, Ashwin and Ruan, Feng and Wainwright, Martin J and Jordan, Michael I},
  journal={arXiv preprint arXiv:2003.07337},
  year={2020}
}

@inproceedings{kearns1999finite,
  title={Finite-sample convergence rates for {Q}-learning and indirect algorithms},
  author={Kearns, Michael J and Singh, Satinder P},
  booktitle={Advances in neural information processing systems},
  pages={996--1002},
  year={1999}
}



@book{bertsekas2010dynamic,
  title={Dynamic Programming and Optimal Control 3rd Edition, Volume II},
  author={Bertsekas, Dimitri P},
  year={2010}
}

@article{kearns2002sparse,
  title={A sparse sampling algorithm for near-optimal planning in large {M}arkov decision processes},
  author={Kearns, Michael and Mansour, Yishay and Ng, Andrew Y},
  journal={Machine learning},
  volume={49},
  number={2-3},
  pages={193--208},
  year={2002},
  publisher={Springer}
}

@book{puterman2014markov,
  title={Markov decision processes: discrete stochastic dynamic programming},
  author={Puterman, Martin L},
  year={2014},
  publisher={John Wiley \& Sons}
}

@book{sutton2018reinforcement,
  title={Reinforcement learning: An introduction},
  author={Sutton, Richard S and Barto, Andrew G},
  year={2018},
  publisher={MIT press}
}

@article{wainwright2019stochastic,
  title={Stochastic approximation with cone-contractive operators: Sharp $\ell_{\infty}$-bounds for {Q}-learning},
  author={Wainwright, Martin J},
  journal={arXiv preprint arXiv:1905.06265},
  year={2019}
}

@article{devraj2020q,
  title={Q-learning with Uniformly Bounded Variance: Large Discounting is Not a Barrier to Fast Learning},
  author={Devraj, Adithya M and Meyn, Sean P},
  journal={arXiv preprint arXiv:2002.10301},
  year={2020}
}

@article{watkins1989learning,
  title={Learning from delayed rewards},
  author={Watkins, Christopher John Cornish Hellaby},
  year={1989},
  publisher={King's College, Cambridge}
}

@article{borkar2000ode,
  title={The {ODE} method for convergence of stochastic approximation and reinforcement learning},
  author={Borkar, Vivek S and Meyn, Sean P},
  journal={SIAM Journal on Control and Optimization},
  volume={38},
  number={2},
  pages={447--469},
  year={2000},
  publisher={SIAM}
}

@inproceedings{sidford2018near,
  title={Near-optimal time and sample complexities for solving {M}arkov decision processes with a generative model},
  author={Sidford, Aaron and Wang, Mengdi and Wu, Xian and Yang, Lin and Ye, Yinyu},
  booktitle={Advances in Neural Information Processing Systems},
  pages={5186--5196},
  year={2018}
}

@inproceedings{sidford2018variance,
  title={Variance reduced value iteration and faster algorithms for solving {M}arkov decision processes},
  author={Sidford, Aaron and Wang, Mengdi and Wu, Xian and Ye, Yinyu},
  booktitle={Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms},
  pages={770--787},
  year={2018},
  organization={SIAM}
}

@article{du2020agnostic,
  title={Agnostic {Q}-learning with Function Approximation in Deterministic Systems: Tight Bounds on Approximation Error and Sample Complexity},
  author={Du, Simon S and Lee, Jason D and Mahajan, Gaurav and Wang, Ruosong},
  journal={arXiv preprint arXiv:2002.07125},
  year={2020}
}

@article{azar2013minimax,
  title={Minimax {PAC} bounds on the sample complexity of reinforcement learning with a generative model},
  author={Azar, Mohammad Gheshlaghi and Munos, R{\'e}mi and Kappen, Hilbert J},
  journal={Machine learning},
  volume={91},
  number={3},
  pages={325--349},
  year={2013},
  publisher={Springer}
}

@book{vershynin2018high,
  title={High-dimensional probability: An introduction with applications in data science},
  author={Vershynin, Roman},
  volume={47},
  year={2018},
  publisher={Cambridge university press}
}

@article{wang2017randomized,
  title={Randomized Linear Programming Solves the {M}arkov Decision Problem in Nearly Linear (Sometimes Sublinear) Time},
  author={Wang, Mengdi},
  journal={Mathematics of Operations Research},
  year={2019},
  publisher={INFORMS}
}

@article{khamaru2021temporal,
  title={Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis},
  author={Khamaru, Koulik and Pananjady, Ashwin and Ruan, Feng and Wainwright, Martin J and Jordan, Michael I},
  journal={SIAM Journal on Mathematics of Data Science},
  volume={3},
  number={4},
  pages={1013--1040},
  year={2021},
  publisher={SIAM}
}

@article{tu2018gap,
  title={The gap between model-based and model-free methods on the linear quadratic regulator: An asymptotic viewpoint},
  author={Tu, Stephen and Recht, Benjamin},
  journal={arXiv preprint arXiv:1812.03565},
  year={2018}
}

@inproceedings{sidford2018near,
  title={Near-optimal time and sample complexities for solving Markov decision processes with a generative model},
  author={Sidford, Aaron and Wang, Mengdi and Wu, Xian and Yang, Lin and Ye, Yinyu},
  booktitle={Advances in Neural Information Processing Systems},
  pages={5186--5196},
  year={2018}
}

@article{agarwal2019optimality,
  title={Model-Based Reinforcement Learning with a Generative Model is Minimax Optimal},
  author={Agarwal, Alekh and Kakade, Sham and Yang, Lin F},
  journal={Conference on Learning Theory},
  pages={67--83},
  year={2020},
}

@article{pearson1901liii,
  title={LIII. On lines and planes of closest fit to systems of points in space},
  author={Pearson, Karl},
  journal={The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science},
  volume={2},
  number={11},
  pages={559--572},
  year={1901},
  publisher={Taylor \& Francis}
}

@article{ding2018leave,
  title={The Leave-one-out Approach for Matrix Completion: Primal and Dual Analysis},
  author={Ding, Lijun and Chen, Yudong},
  journal={arXiv preprint arXiv:1803.07554},
  year={2018}
}

@inproceedings{cai2019nonconvex,
  title={Nonconvex Low-Rank Symmetric Tensor Completion from Noisy Data},
  author={Cai, Changxiao and Li, Gen and Poor, H Vincent and Chen, Yuxin},
  booktitle={Advances in Neural Information Processing Systems},
  pages={1861--1872},
  year={2019}
}


@inproceedings{foygel2011concentration,
  title={Concentration-based guarantees for low-rank matrix reconstruction},
  author={Foygel, Rina and Srebro, Nathan},
  booktitle={Conference on Learning Theory},
  pages={315--340},
  year={2011}
}

@inproceedings{jalali2011clustering,
  title={Clustering Partially Observed Graphs via Convex Optimization.},
  author={Jalali, Ali and Chen, Yudong and Sanghavi, Sujay and Xu, Huan},
  booktitle={International Conference on Machine Learning},
  volume={11},
  pages={1001--1008},
  year={2011}
}

@inproceedings{chen2012clustering,
  title={Clustering sparse graphs},
  author={Chen, Yudong and Sanghavi, Sujay and Xu, Huan},
  booktitle={Advances in neural information processing systems},
  pages={2204--2212},
  year={2012}
}

@book{wainwright2019high,
  title={High-Dimensional Statistics: A Non-Asymptotic Viewpoint},
  author={Wainwright, M.J.},
  series={Cambridge Series in Statistical and Probabilistic Mathematics},
  year={2019},
  publisher={Cambridge University Press}
}

@article{laurent2000adaptive,
  title={Adaptive estimation of a quadratic functional by model selection},
  author={Laurent, Beatrice and Massart, Pascal},
  journal={Annals of Statistics},
  pages={1302--1338},
  year={2000},
  publisher={JSTOR}
}



@article{carpentier2018adaptive,
  title={Adaptive confidence sets for matrix completion},
  author={Carpentier, Alexandra and Klopp, Olga and L{\"o}ffler, Matthias and Nickl, Richard},
  journal={Bernoulli},
  volume={24},
  number={4A},
  pages={2429--2460},
  year={2018},
  publisher={Bernoulli Society for Mathematical Statistics and Probability}
}


@inproceedings{sun2012calibrated,
  title={Calibrated elastic regularization in matrix completion},
  author={Sun, Tingni and Zhang, Cun-Hui},
  booktitle={Advances in Neural Information Processing Systems},
  pages={863--871},
  year={2012}
}


@article{berger1989estimating,
  title={Estimating a product of means: Bayesian analysis with reference priors},
  author={Berger, James O and Bernardo, Jos{\'e} M},
  journal={Journal of the American Statistical Association},
  volume={84},
  number={405},
  pages={200--207},
  year={1989},
  publisher={Taylor \& Francis}
}

@article{efron1986isn,
  title={Why isn't everyone a Bayesian?},
  author={Efron, Bradley},
  journal={The American Statistician},
  volume={40},
  number={1},
  pages={1--5},
  year={1986},
  publisher={Taylor \& Francis Group}
}

@article{bentkus2005lyapunov,
  title={A Lyapunov-type bound in Rd},
  author={Bentkus, Vidmantas},
  journal={Theory of Probability \& Its Applications},
  volume={49},
  number={2},
  pages={311--323},
  year={2005},
  publisher={SIAM}
}

@article{smith1968matrix,
  title={Matrix Equation ${XA+BX=C}$},
  author={Smith, RA},
  journal={SIAM Journal on Applied Mathematics},
  volume={16},
  number={1},
  pages={198--201},
  year={1968},
  publisher={SIAM}
}

@article{cai2013max,
  title={A max-norm constrained minimization approach to 1-bit matrix completion},
  author={Cai, Tony and Zhou, Wen-Xin},
  journal={The Journal of Machine Learning Research},
  volume={14},
  number={1},
  pages={3619--3647},
  year={2013},
  publisher={JMLR. org}
}


@article{cai2013sparse,
  title={Sparse {PCA}: Optimal rates and adaptive estimation},
  author={Cai, T Tony and Ma, Zongming and Wu, Yihong},
  journal={The Annals of Statistics},
  volume={41},
  number={6},
  pages={3074--3110},
  year={2013},
  publisher={Institute of Mathematical Statistics}
}

@article{zou2006sparse,
  title={Sparse principal component analysis},
  author={Zou, Hui and Hastie, Trevor and Tibshirani, Robert},
  journal={Journal of computational and graphical statistics},
  volume={15},
  number={2},
  pages={265--286},
  year={2006},
  publisher={Taylor \& Francis}
}

@inproceedings{d2005direct,
  title={A direct formulation for sparse PCA using semidefinite programming},
  author={d'Aspremont, Alexandre and Ghaoui, Laurent E and Jordan, Michael I and Lanckriet, Gert R},
  booktitle={Advances in neural information processing systems},
  pages={41--48},
  year={2005}
}


@article{stewart1977perturbation,
  title={On the perturbation of pseudo-inverses, projections and linear least squares problems},
  author={Stewart, Gilbert W},
  journal={SIAM review},
  volume={19},
  number={4},
  pages={634--662},
  year={1977},
  publisher={SIAM}
}

@inproceedings{carpentier2016constructing,
  title={Constructing confidence sets for the matrix completion problem},
  author={Carpentier, Alexandra and Klopp, Olga and L{\"o}ffler, Matthias},
  booktitle={Conference of the International Society for Non-Parametric Statistics},
  pages={103--118},
  year={2016},
  organization={Springer}
}

@article{koltchinskii2015optimal,
  title={Optimal estimation of low rank density matrices.},
  author={Koltchinskii, Vladimir and Xia, Dong},
  journal={Journal of Machine Learning Research},
  volume={16},
  pages={1757--1792},
  year={2015}
}

@article{fan2019asymptotic,
  title={Asymptotic Theory of Eigenvectors for Large Random Matrices},
  author={Fan, Jianqing and Fan, Yingying and Han, Xiao and Lv, Jinchi},
  journal={arXiv preprint arXiv:1902.06846},
  year={2019}
}

@article{ling2017blind,
  title={Blind deconvolution meets blind demixing: Algorithms and performance bounds},
  author={Ling, Shuyang and Strohmer, Thomas},
  journal={IEEE Transactions on Information Theory},
  volume={63},
  number={7},
  pages={4497--4520},
  year={2017},
  publisher={IEEE}
}

@article{mak2017active,
  title={Active matrix completion with uncertainty quantification},
  author={Mak, Simon and Xie, Yao},
  journal={arXiv preprint arXiv:1706.08037},
  year={2017}
}

@article{zhang2014confidence,
  title={Confidence intervals for low dimensional parameters in high dimensional linear models},
  author={Zhang, Cun-Hui and Zhang, Stephanie S},
  journal={Journal of the Royal Statistical Society: Series B},
  volume={76},
  number={1},
  pages={217--242},
  year={2014},
  publisher={Wiley Online Library}
}

@article{cai2017confidence,
  title={Confidence intervals for high-dimensional linear regression: Minimax rates and adaptivity},
  author={Cai, T Tony and Guo, Zijian},
  journal={The Annals of statistics},
  volume={45},
  number={2},
  pages={615--646},
  year={2017},
  publisher={Institute of Mathematical Statistics}
}

@article{ning2017general,
  title={A general theory of hypothesis tests and confidence regions for sparse high dimensional models},
  author={Ning, Yang and Liu, Han},
  journal={The Annals of Statistics},
  volume={45},
  number={1},
  pages={158--195},
  year={2017},
  publisher={Institute of Mathematical Statistics}
}

@article{javanmard2014confidence,
  title={Confidence intervals and hypothesis testing for high-dimensional regression},
  author={Javanmard, Adel and Montanari, Andrea},
  journal={The Journal of Machine Learning Research},
  volume={15},
  number={1},
  pages={2869--2909},
  year={2014},
  publisher={JMLR. org}
}

@article{jankova2015confidence,
  title={Confidence intervals for high-dimensional inverse covariance estimation},
  author={Jankova, Jana and Van De Geer, Sara},
  journal={Electronic Journal of Statistics},
  volume={9},
  number={1},
  pages={1205--1229},
  year={2015},
  publisher={The Institute of Mathematical Statistics and the Bernoulli Society}
}

@article{van2014asymptotically,
  title={On asymptotically optimal confidence regions and tests for high-dimensional models},
  author={van de Geer, Sara and B{\"u}hlmann, Peter and Ritov, Ya'acov and Dezeure, Ruben},
  journal={The Annals of Statistics},
  volume={42},
  number={3},
  pages={1166--1202},
  year={2014},
  publisher={Institute of Mathematical Statistics}
}

@article{chen2019noisy,
  title={Noisy Matrix Completion: Understanding Statistical Guarantees for Convex Relaxation via Nonconvex Optimization},
  author={Chen, Yuxin and Chi, Yuejie and Fan, Jianqing and Ma, Cong and Yan, Yuling},
  journal={arXiv:1902.07698},
  year={2019}
}

@article{chen2019nonconvex,
  title={Nonconvex Rectangular Matrix Completion via Gradient Descent without $\ell_{2,\infty}$ Regularization},
  author={Ji Chen and Dekai Liu and Xiaodong Li},
  journal={arXiv:1901.06116v1},
  year={2019}
}


@inproceedings{gunasekar2013noisy,
  title={Noisy matrix completion using alternating minimization},
  author={Gunasekar, Suriya and Acharya, Ayan and Gaur, Neeraj and Ghosh, Joydeep},
  booktitle={Joint European Conference on Machine Learning and Knowledge Discovery in Databases},
  pages={194--209},
  year={2013},
}


@article{chen2018gradient,
	Author = {Chen, Yuxin and Chi, Yuejie and Fan, Jianqing and Ma, Cong},
	Journal = {Mathematical Programming},
	Title = {Gradient Descent with Random Initialization: Fast Global Convergence for Nonconvex Phase Retrieval},
	Year = {2019},
  	month = {July},
	volume = {176},
	number = {1-2},
	pages = {5-37}
}

@article{cai2016matrix,
  title={Matrix completion via max-norm constrained optimization},
  author={Cai, T Tony and Zhou, Wen-Xin},
  journal={Electronic Journal of Statistics},
  volume={10},
  number={1},
  pages={1493--1525},
  year={2016},
  publisher={The Institute of Mathematical Statistics and the Bernoulli Society}
}

@article{klopp2014noisy,
  title={Noisy low-rank matrix completion with general sampling distribution},
  author={Klopp, Olga},
  journal={Bernoulli},
  volume={20},
  number={1},
  pages={282--303},
  year={2014},
  publisher={Bernoulli Society for Mathematical Statistics and Probability}
}

@article{nesterov2012make,
  title={How to make the gradients small},
  author={Nesterov, Yurii},
  journal={Optima},
  volume={88},
  pages={10--11},
  year={2012}
}

@inproceedings{yi2016fast,
  title={Fast algorithms for robust {PCA} via gradient descent},
  author={Yi, Xinyang and Park, Dohyung and Chen, Yudong and Caramanis, Constantine},
  booktitle={NIPS},
  pages={4152--4160},
  year={2016}
}

@inproceedings{dong2018nonconvex,
  title={Nonconvex demixing from bilinear measurements},
  author={Dong, Jialin and Shi, Yuanming},
  booktitle={IEEE Transactions on Signal Processing},
  pages={5152--5166},
  year={2018},
}

@inproceedings{zhou2010stable,
  title={Stable principal component pursuit},
  author={Zhou, Z. and Li, X. and Wright, J. and Cand\`es, E. and Ma, Y.},
  booktitle={International Symposium on Information Theory},
  pages={1518--1522},
  year={2010},
}

@inproceedings{rajkumar2016can,
  title={When can we rank well from comparisons of O (n$\backslash$log (n)) non-actively chosen pairs?},
  author={Rajkumar, Arun and Agarwal, Shivani},
  booktitle={Conference on Learning Theory},
  pages={1376--1401},
  year={2016}
}

@book{boyd2004convex,
  title={Convex optimization},
  author={Boyd, Stephen and Vandenberghe, Lieven},
  year={2004},
  publisher={Cambridge university press}
}

@article{chen2013low,
  title={Low-rank matrix recovery from errors and erasures},
  author={Chen, Yudong and Jalali, Ali and Sanghavi, Sujay and Caramanis, Constantine},
  journal={IEEE Transactions on Information Theory},
  volume={59},
  number={7},
  pages={4324--4337},
  year={2013},
  publisher={IEEE}
}

@article{parikh2014proximal,
  title={Proximal algorithms},
  author={Parikh, Neal and Boyd, Stephen},
  journal={Foundations and Trends{\textregistered} in Optimization},
  volume={1},
  number={3},
  pages={127--239},
  year={2014},
}

@article{cai2019fast,
  title={Fast and provable algorithms for spectrally sparse signal reconstruction via low-rank Hankel matrix completion},
  author={Cai, Jian-Feng and Wang, Tianming and Wei, Ke},
  journal={Applied and Computational Harmonic Analysis},
  volume={46},
  number={1},
  pages={94--121},
  year={2019},
  publisher={Elsevier}
}

@article{fazel2013hankel,
  title={Hankel matrix rank minimization with applications to system identification and realization},
  author={Fazel, Maryam and Pong, Ting Kei and Sun, Defeng and Tseng, Paul},
  journal={SIAM Journal on Matrix Analysis and Applications},
  volume={34},
  number={3},
  pages={946--977},
  year={2013},
  publisher={SIAM}
}

@article{cai2016robust,
  title={Robust recovery of complex exponential signals from random Gaussian projections via low rank Hankel matrix reconstruction},
  author={Cai, Jian-Feng and Qu, Xiaobo and Xu, Weiyu and Ye, Gui-Bo},
  journal={Applied and computational harmonic analysis},
  volume={41},
  number={2},
  pages={470--490},
  year={2016},
  publisher={Elsevier}
}

@article{natarajan1995sparse,
  title={Sparse approximate solutions to linear systems},
  author={Natarajan, Balas Kausik},
  journal={SIAM journal on computing},
  volume={24},
  number={2},
  pages={227--234},
  year={1995},
  publisher={SIAM}
}

@inproceedings{candes2014mathematics,
  title={Mathematics of sparsity (and a few other things)},
  author={Cand{\`e}s, Emmanuel},
  booktitle={Proceedings of the International Congress of Mathematicians, Seoul, South Korea},
  volume={123},
  year={2014},
}

@inproceedings{srebro2005rank,
  title={Rank, trace-norm and max-norm},
  author={Srebro, Nathan and Shraibman, Adi},
  booktitle={International Conference on Computational Learning Theory},
  pages={545--560},
  year={2005},
  organization={Springer}
}


@article{liu2009interior,
  title={Interior-point method for nuclear norm approximation with application to system identification},
  author={Liu, Zhang and Vandenberghe, Lieven},
  journal={SIAM Journal on Matrix Analysis and Applications},
  volume={31},
  number={3},
  pages={1235--1256},
  year={2009},
  publisher={SIAM}
}

@book{Bos12,
	title={Linear processes in function spaces: theory and applications},
	author={Bosq, Denis},
	volume={149},
	year={2012},
	publisher={Springer Science \& Business Media}
}

@article{so2007theory,
  title={Theory of semidefinite programming for sensor network localization},
  author={So, Anthony Man-Cho and Ye, Yinyu},
  journal={Mathematical Programming},
  volume={109},
  number={2-3},
  pages={367--384},
  year={2007},
  publisher={Springer}
}


@article{ma2018optimization,
  title={Optimization-based AMP for Phase Retrieval: The Impact of Initialization and $\ell\_2 $-regularization},
  author={Ma, Junjie and Xu, Ji and Maleki, Arian},
  journal={arXiv preprint arXiv:1801.01170},
  year={2018}
}

@article{li2018nonconvex,
  title={Nonconvex Matrix Factorization from Rank-One Measurements},
  author={Li, Yuanxin and Ma, Cong and Chen, Yuxin and Chi, Yuejie},
  journal={arXiv:1802.06286, accepted to AISTATS},
  year={2018}
}

@article{fu2018local,
  title={Local Geometry of One-Hidden-Layer Neural Networks for Logistic Regression},
  author={Fu, Haoyu and Chi, Yuejie and Liang, Yingbin},
  journal={arXiv preprint arXiv:1802.06463},
  year={2018}
}

@article{murray2017revisiting,
  title={Revisiting Normalized Gradient Descent: Evasion of Saddle Points},
  author={Murray, Ryan and Swenson, Brian and Kar, Soummya},
  journal={arXiv preprint arXiv:1711.05224},
  year={2017}
}



@inproceedings{ge2015escaping,
  title={Escaping from saddle points online stochastic gradient for tensor decomposition},
  author={Ge, Rong and Huang, Furong and Jin, Chi and Yuan, Yang},
  booktitle={Conference on Learning Theory},
  pages={797--842},
  year={2015}
}

@article{zhang2017phase,
  title={Phase retrieval using alternating minimization in a batch setting},
  author={Zhang, Teng},
  journal={arXiv preprint arXiv:1706.08167},
  year={2017}
}

@article{zhang2017compressive,
  title={Compressive Phase Retrieval via Reweighted Amplitude Flow},
  author={Zhang, Liang and Wang, Gang and Giannakis, Georgios B and Chen, Jie},
  journal={arXiv preprint arXiv:1712.02426},
  year={2017}
}

@article{li2017linear,
  title={Linear Convergence of An Iterative Phase Retrieval Algorithm with Data Reuse},
  author={Li, Gen and Jiao, Yuchen and Gu, Yuantao},
  journal={arXiv preprint arXiv:1712.01712},
  year={2017}
}

@article{soltanolkotabi2017theoretical,
  title={Theoretical insights into the optimization landscape of over-parameterized shallow neural networks},
  author={Soltanolkotabi, Mahdi and Javanmard, Adel and Lee, Jason D},
  journal={arXiv preprint arXiv:1707.04926},
  year={2017}
}

@article{li2017algorithmic,
  title={Algorithmic Regularization in Over-parameterized Matrix Recovery},
  author={Li, Yuanzhi and Ma, Tengyu and Zhang, Hongyang},
  journal={arXiv preprint arXiv:1712.09203},
  year={2017}
}

@misc{vershynin2016high,
  title={High Dimensional Probability},
  author={Vershynin, Roman},
  publisher={Cambridge University Press},
  year={2017}
}

@article{lee2017first,
  title={First-order Methods Almost Always Avoid Saddle Points},
  author={Lee, Jason D and Panageas, Ioannis and Piliouras, Georgios and Simchowitz, Max and Jordan, Michael I and Recht, Benjamin},
  journal={arXiv preprint arXiv:1710.07406},
  year={2017}
}


@article{jin2017escape,
  title={How to escape saddle points efficiently},
  author={Jin, Chi and Ge, Rong and Netrapalli, Praneeth and Kakade, Sham M and Jordan, Michael I},
  journal={arXiv preprint arXiv:1703.00887},
  year={2017}
}

@inproceedings{du2017gradient,
  title={Gradient descent can take exponential time to escape saddle points},
  author={Du, Simon S and Jin, Chi and Lee, Jason D and Jordan, Michael I and Singh, Aarti and Poczos, Barnabas},
  booktitle={Advances in Neural Information Processing Systems},
  pages={1067--1077},
  year={2017}
}

@article{lee2016gradient,
  title={Gradient descent converges to minimizers},
  author={Lee, Jason D and Simchowitz, Max and Jordan, Michael I and Recht, Benjamin},
  journal={arXiv preprint arXiv:1602.04915},
  year={2016}
}

@article{jin2017accelerated,
  title={Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent},
  author={Jin, Chi and Netrapalli, Praneeth and Jordan, Michael I},
  journal={arXiv preprint arXiv:1711.10456},
  year={2017}
}


@article{shechtman2015phase,
  title={Phase retrieval with application to optical imaging: a contemporary overview},
  author={Shechtman, Yoav and Eldar, Yonina C and Cohen, Oren and Chapman, Henry Nicholas and Miao, Jianwei and Segev, Mordechai},
  journal={IEEE signal processing magazine},
  volume={32},
  number={3},
  pages={87--109},
  year={2015},
  publisher={IEEE}
}

@incollection{mixon2015phase,
  title={Phase transitions in phase retrieval},
  author={Mixon, Dustin G},
  booktitle={Excursions in Harmonic Analysis, Volume 4},
  pages={123--147},
  year={2015},
  publisher={Springer}
}

@book{jolliffe2011principal,
  title={Principal component analysis},
  author={Jolliffe, Ian},
  year={2011},
  publisher={Springer}
}

@article{fan2018principal,
  title={Principal component analysis for big data},
  author={Fan, Jianqing and Sun, Qiang and Zhou, Wen-Xin and Zhu, Ziwei},
  journal={arXiv preprint arXiv:1801.01602},
  year={2018}
}

@inproceedings{jin2016provable,
  title={Provable efficient online matrix completion via non-convex stochastic gradient descent},
  author={Jin, Chi and Kakade, Sham M and Netrapalli, Praneeth},
  booktitle={NIPS},
  pages={4520--4528},
  year={2016}
}

@article{ma2017implicit,
  title={Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion and Blind Deconvolution},
  author={Ma, Cong and Wang, Kaizheng and Chi, Yuejie and Chen, Yuxin},
  journal={Foundations of Computational Mathematics},
  volume = {20},
  number = {3},
  pages = {451-632},
  year={2020}
}

@article{chi2018nonconvex,
  title={Nonconvex optimization meets low-rank matrix factorization: An overview},
  author={Chi, Yuejie and Lu, Yue M and Chen, Yuxin},
  journal={IEEE Transactions on Signal Processing},
  volume = {67},
  number = {20},
  pages = {5239 - 5269}, 
  month = {October},
  year={2019}
}


@article{dulac2019challenges,
  title={Challenges of real-world reinforcement learning},
  author={Dulac-Arnold, Gabriel and Mankowitz, Daniel and Hester, Todd},
  journal={arXiv preprint arXiv:1904.12901},
  year={2019}
}

@ARTICLE{chen2018harnessing,
author={Yudong Chen and Yuejie Chi},
journal={IEEE Signal Processing Magazine},
title={Harnessing Structures in Big Data via Guaranteed Low-Rank Matrix Estimation: Recent Theory and Fast Algorithms via Convex and Nonconvex Optimization},
year={2018},
volume={35},
number={4},
pages={14-31},
keywords={Big Data;concave programming;convex programming;learning (artificial intelligence);matrix algebra;minimisation;big data;guaranteed low-rank matrix estimation;nonconvex optimization;low-rank modeling;signal processing;machine learning;collaborative filtering;video surveillance;medical imaging;dimensionality reduction;adaptive filtering;high-dimensional data;low-dimensional subspace;low-rank structures;computationally efficient procedures;nonconvex approaches;convex relaxations;nuclear norm minimization;statistically optimal procedures;low-rank matrices;computational challenges;nonconvex procedures;globally optimal solutions;algorithmic designs;cost reduction;structural properties;Estimation;Signal processing algorithms;Data models;Compressed sensing;Sparse matrices;Optimization;Signal processing},
doi={10.1109/MSP.2018.2821706},
ISSN={1053-5888},
month={July},}


@article{davenport2016overview,
  title={An overview of low-rank matrix recovery from incomplete observations},
  author={Davenport, Mark A and Romberg, Justin},
  journal={IEEE Journal of Selected Topics in Signal Processing},
  volume={10},
  number={4},
  pages={608--622},
  year={2016},
  publisher={IEEE}
}

@article{kreutz2009complex,
  title={The complex gradient operator and the {CR}-calculus},
  author={Kreutz-Delgado, Ken},
  journal={arXiv preprint arXiv:0906.4835},
  year={2009}
}

@article{lerman2014fast,
  title={Fast, robust and non-convex subspace recovery},
  author={Lerman, Gilad and Maunu, Tyler},
  journal={arXiv preprint arXiv:1406.6145},
  year={2014}
}

@article{keskar2016large,
  title={On large-batch training for deep learning: Generalization gap and sharp minima},
  author={Keskar, Nitish Shirish and Mudigere, Dheevatsa and Nocedal, Jorge and Smelyanskiy, Mikhail and Tang, Ping Tak Peter},
  journal={arXiv preprint arXiv:1609.04836},
  year={2016}
}

@article{soudry2017implicit,
  title={The Implicit Bias of Gradient Descent on Separable Data},
  author={Soudry, Daniel and Hoffer, Elad and Srebro, Nathan},
  journal={arXiv preprint arXiv:1710.10345},
  year={2017}
}

@book{Koltchinskii2011oracle,
    AUTHOR = {Koltchinskii, Vladimir},
     TITLE = {Oracle inequalities in empirical risk minimization and sparse
              recovery problems},
    SERIES = {Lecture Notes in Mathematics},
    VOLUME = {2033},
 PUBLISHER = {Springer, Heidelberg},
      YEAR = {2011},
     PAGES = {x+254},
      ISBN = {978-3-642-22146-0},
   MRCLASS = {91B30 (60B11 62G08 62J07 68T05)},
  MRNUMBER = {2829871},
MRREVIEWER = {Alexandre B. Tsybakov},
       URL = {https://doi.org/10.1007/978-3-642-22147-7},
}

@article{wei2015solving,
  title={Solving systems of phaseless equations via {K}aczmarz methods: A proof of concept study},
  author={Wei, Ke},
  journal={Inverse Problems},
  volume={31},
  number={12},
  pages={125008},
  year={2015},
  publisher={IOP Publishing}
}

@article{wei2016guarantees,
  title={Guarantees of {R}iemannian optimization for low rank matrix recovery},
  author={Wei, K. and Cai, J.F. and Chan, T. and Leung, S.},
  journal={SIAM Journal on Matrix Analysis and Applications},
  volume={37},
  number={3},
  pages={1198--1222},
  year={2016},
  publisher={SIAM}
}

@inproceedings{zhang2017global,
  title={On the Global Geometry of Sphere-Constrained Sparse Blind Deconvolution},
  author={Zhang, Yuqian and Lau, Yenson and Kuo, Han-wen and Cheung, Sky and Pasupathy, Abhay and Wright, John},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={4894--4902},
  year={2017}
}

@article{chi2016guaranteed,
  title={Guaranteed blind sparse spikes deconvolution via lifting and convex optimization},
  author={Chi, Yuejie},
  journal={IEEE Journal of Selected Topics in Signal Processing},
  volume={10},
  number={4},
  pages={782--794},
  year={2016},
  publisher={IEEE}
}

@article{ling2017regularized,
  title={Regularized Gradient Descent: A Nonconvex Recipe for Fast Joint Blind Deconvolution and Demixing},
  author={Ling, Shuyang and Strohmer, Thomas},
  journal={arXiv preprint arXiv:1703.08642},
  year={2017}
}

@article{aghasi2017branchhull,
  title={BranchHull: Convex bilinear inversion from the entrywise product of signals with known signs},
  author={Aghasi, Alireza and Ahmed, Ali and Hand, Paul},
  journal={arXiv preprint arXiv:1702.04342},
  year={2017}
}

@article{lee2017blind,
  title={Blind recovery of sparse signals from subsampled convolution},
  author={Lee, Kiryung and Li, Yanjun and Junge, Marius and Bresler, Yoram},
  journal={IEEE Transactions on Information Theory},
  volume={63},
  number={2},
  pages={802--821},
  year={2017},
  publisher={IEEE}
}

@article{ling2015self,
  title={Self-calibration and biconvex compressive sensing},
  author={Ling, Shuyang and Strohmer, Thomas},
  journal={Inverse Problems},
  volume={31},
  number={11},
  pages={115002},
  year={2015},
  publisher={IOP Publishing}
}

@article{javanmard2014hypothesis,
  title={Hypothesis testing in high-dimensional regression under the gaussian random design model: Asymptotic theory},
  author={Javanmard, Adel and Montanari, Andrea},
  journal={IEEE Transactions on Information Theory},
  volume={60},
  number={10},
  pages={6522--6554},
  year={2014},
  publisher={IEEE}
}

@article{meinshausen2009p,
  title={P-values for high-dimensional regression},
  author={Meinshausen, Nicolai and Meier, Lukas and B{\"u}hlmann, Peter},
  journal={Journal of the American Statistical Association},
  volume={104},
  number={488},
  pages={1671--1681},
  year={2009},
  publisher={Taylor \& Francis}
}

@article{zhang2017simultaneous,
  title={Simultaneous inference for high-dimensional linear models},
  author={Zhang, Xianyang and Cheng, Guang},
  journal={Journal of the American Statistical Association},
  volume={112},
  number={518},
  pages={757--768},
  year={2017},
  publisher={Taylor \& Francis}
}

@article{charisopoulos2019composite,
  title={Composite optimization for robust blind deconvolution},
  author={Charisopoulos, Vasileios and Davis, Damek and D{\'\i}az, Mateo and Drusvyatskiy, Dmitriy},
  journal={arXiv preprint arXiv:1901.01624},
  year={2019}
}

@article{charisopoulos2019low,
  title={Low-rank matrix recovery with composite optimization: good conditioning and rapid convergence},
  author={Charisopoulos, Vasileios and Chen, Yudong and Davis, Damek and D{\'\i}az, Mateo and Ding, Lijun and Drusvyatskiy, Dmitriy},
  journal={arXiv preprint arXiv:1904.10020},
  year={2019}
}

@article{xia2018confidence,
  title={Confidence interval of singular vectors for high-dimensional and low-rank matrix regression},
  author={Xia, Dong},
  journal={arXiv preprint arXiv:1805.09871},
  year={2018}
}

@article{belloni2011inference,
  title={Inference for high-dimensional sparse econometric models},
  author={Belloni, Alexandre and Chernozhukov, Victor and Hansen, Christian},
  journal={arXiv preprint arXiv:1201.0220},
  year={2011}
}

@article{jankova2018biased,
  title={De-biased sparse PCA: Inference and testing for eigenstructure of large covariance matrices},
  author={Jankov{\'a}, Jana and van de Geer, Sara},
  journal={arXiv preprint arXiv:1801.10567},
  year={2018}
}

@article{cai2016geometric,
  title={Geometric inference for general high-dimensional linear inverse problems},
  author={Cai, T Tony and Liang, Tengyuan and Rakhlin, Alexander},
  journal={The Annals of Statistics},
  volume={44},
  number={4},
  pages={1536--1563},
  year={2016},
  publisher={Institute of Mathematical Statistics}
}

@article{xia2019data,
  title={Data-dependent Confidence Regions of Singular Subspaces},
  author={Xia, Dong},
  journal={arXiv preprint arXiv:1901.00304},
  year={2019}
}

@article{carpentier2015uncertainty,
  title={Uncertainty quantification for matrix compressed sensing and quantum tomography problems},
  author={Carpentier, Alexandra and Eisert, Jens and Gross, David and Nickl, Richard},
  journal={arXiv preprint arXiv:1504.03234},
  year={2015}
}

@article{jankova2017honest,
  title={Honest confidence regions and optimality in high-dimensional precision matrix estimation},
  author={Jankov{\'a}, Jana and van de Geer, Sara},
  journal={Test},
  volume={26},
  number={1},
  pages={143--162},
  year={2017},
  publisher={Springer}
}



@article{davenport20141,
  title={1-bit matrix completion},
  author={Davenport, Mark A and Plan, Yaniv and {Van Den Berg}, Ewout and Wootters, Mary},
  journal={Information and Inference},
  volume={3},
  number={3},
  pages={189--223},
  year={2014},
  publisher={OUP}
}

@article{neykov2018unified,
  title={A unified theory of confidence regions and testing for high-dimensional estimating equations},
  author={Neykov, Matey and Ning, Yang and Liu, Jun S and Liu, Han},
  journal={Statistical Science},
  volume={33},
  number={3},
  pages={427--443},
  year={2018},
  publisher={Institute of Mathematical Statistics}
}

@inproceedings{zhang2018primal,
  title={A primal-dual analysis of global optimality in nonconvex low-rank matrix recovery},
  author={Zhang, Xiao and Wang, Lingxiao and Yu, Yaodong and Gu, Quanquan},
  booktitle={International conference on machine learning},
  pages={5857--5866},
  year={2018}
}

@misc{chernozhukov2018double,
  title={Double/debiased machine learning for treatment and structural parameters},
  author={Chernozhukov, Victor and Chetverikov, Denis and Demirer, Mert and Duflo, Esther and Hansen, Christian and Newey, Whitney and Robins, James},
  year={2018},
  publisher={Oxford University Press Oxford, UK}
}

@article{srebro2004learning,
  title={Learning with matrix factorizations},
  journal = {Ph.~D.~thesis},
  author={Srebro, Nathan},
  year={2004}
}

@article{ren2015asymptotic,
  title={Asymptotic normality and optimalities in estimation of large Gaussian graphical models},
  author={Ren, Z. and Sun, T. and Zhang, C. and Zhou, H.},
  journal={The Annals of Statistics},
  volume={43},
  number={3},
  pages={991--1026},
  year={2015},
  publisher={Institute of Mathematical Statistics}
}

@article{wasserman2009high,
  title={High dimensional variable selection},
  author={Wasserman, Larry and Roeder, Kathryn},
  journal={Annals of statistics},
  volume={37},
  number={5A},
  pages={2178},
  year={2009},
  publisher={NIH Public Access}
}

@article{dezeure2017high,
  title={High-dimensional simultaneous inference with the bootstrap},
  author={Dezeure, Ruben and B{\"u}hlmann, Peter and Zhang, Cun-Hui},
  journal={Test},
  volume={26},
  number={4},
  pages={685--719},
  year={2017},
  publisher={Springer}
}

@article{athey2018approximate,
  title={Approximate residual balancing: debiased inference of average treatment effects in high dimensions},
  author={Athey, Susan and Imbens, Guido W and Wager, Stefan},
  journal={Journal of the Royal Statistical Society: Series B (Statistical Methodology)},
  volume={80},
  number={4},
  pages={597--623},
  year={2018},
  publisher={Wiley Online Library}
}

@article{dezeure2015high,
  title={High-dimensional inference: Confidence intervals, p-values and r-software hdi},
  author={Dezeure, R. and B{\"u}hlmann, P. and Meier, L. and Meinshausen, N.},
  journal={Statistical science},
  pages={533--558},
  year={2015},
  publisher={JSTOR}
}

@article{lee2016exact,
  title={Exact post-selection inference, with application to the lasso},
  author={Lee, Jason D and Sun, Dennis L and Sun, Yuekai and Taylor, Jonathan E},
  journal={The Annals of Statistics},
  volume={44},
  number={3},
  pages={907--927},
  year={2016},
  publisher={Institute of Mathematical Statistics}
}

@article{lockhart2014significance,
  title={A significance test for the lasso},
  author={Lockhart, Richard and Taylor, Jonathan and Tibshirani, Ryan J and Tibshirani, Robert},
  journal={Annals of statistics},
  volume={42},
  number={2},
  pages={413},
  year={2014},
  publisher={NIH Public Access}
}

@article {Negahban2012rscMC,
    AUTHOR = {Negahban, Sahand and Wainwright, Martin J.},
     TITLE = {Restricted strong convexity and weighted matrix completion:
              optimal bounds with noise},
   JOURNAL = {J. Mach. Learn. Res.},
  FJOURNAL = {Journal of Machine Learning Research (JMLR)},
    VOLUME = {13},
      YEAR = {2012},
     PAGES = {1665--1697},
      ISSN = {1532-4435},
   MRCLASS = {62H12 (15A83 60B20 60E15 62G20)},
  MRNUMBER = {2930649},
}

@article{krahmer2019convex,
  title={On the convex geometry of blind deconvolution and matrix completion},
  author={Krahmer, Felix and St{\"o}ger, Dominik},
  journal={arXiv preprint arXiv:1902.11156},
  year={2019}
}

@article{singer2011angular,
  title={Angular synchronization by eigenvectors and semidefinite programming},
  author={Singer, Amit},
  journal={Applied and computational harmonic analysis},
  volume={30},
  number={1},
  pages={20--36},
  year={2011},
  publisher={Elsevier}
}

@article{lee2016fast,
  title={Fast and guaranteed blind multichannel deconvolution under a bilinear system model},
  author={Lee, Kiryung and Tian, Ning and Romberg, Justin},
  journal={arXiv preprint arXiv:1610.06469},
  year={2016}
}

@article {Se2010Noisy,
    AUTHOR = {Keshavan, Raghunandan H. and Montanari, Andrea and Oh,
              Sewoong},
     TITLE = {Matrix completion from noisy entries},
   JOURNAL = {J. Mach. Learn. Res.},
  FJOURNAL = {Journal of Machine Learning Research (JMLR)},
    VOLUME = {11},
      YEAR = {2010},
     PAGES = {2057--2078},
      ISSN = {1532-4435},
   MRCLASS = {62H12 (15A83 62J99)},
  MRNUMBER = {2678022},
}

@article{dhifallah2017fundamental,
  title={Fundamental limits of PhaseMax for phase retrieval: A replica analysis},
  author={Dhifallah, Oussama and Lu, Yue M},
  journal={arXiv preprint arXiv:1708.03355},
  year={2017}
}

@article{wang2016blind,
  title={Blind deconvolution from multiple sparse inputs},
  author={Wang, Liming and Chi, Yuejie},
  journal={IEEE Signal Processing Letters},
  volume={23},
  number={10},
  pages={1384--1388},
  year={2016},
  publisher={IEEE}
}


@article{dhifallah2017phase,
  title={Phase Retrieval via Linear Programming: Fundamental Limits and Algorithmic Improvements},
  author={Dhifallah, Oussama and Thrampoulidis, Christos and Lu, Yue M},
  journal={arXiv preprint arXiv:1710.05234},
  year={2017}
}

@article{hand2016elementary,
  title={An elementary proof of convex phase retrieval in the natural parameter space via the linear program {P}hase{M}ax},
  author={Hand, Paul and Voroninski, Vladislav},
  journal={arXiv preprint arXiv:1611.03935},
  year={2016}
}

@incollection{tropp2015convex,
  title={Convex recovery of a structured signal from independent random linear measurements},
  author={Tropp, Joel A},
  booktitle={Sampling Theory, a Renaissance},
  pages={67--101},
  year={2015},
  publisher={Springer}
}

@article{soltanolkotabi2017structured,
  title={Structured signal recovery from quadratic measurements: Breaking sample complexity barriers via nonconvex optimization},
  author={Soltanolkotabi, Mahdi},
  journal={arXiv preprint arXiv:1702.06175},
  year={2017}
}


@inproceedings{bahmani2016phase,
  title={Phase Retrieval Meets Statistical Learning Theory: A Flexible Convex Relaxation},
  author={Bahmani, Sohail and Romberg, Justin},
  booktitle={Artificial Intelligence and Statistics},
  pages={252--260},
  year={2017}
}

@article{liu2017algorithmic,
  title={Algorithmic stability and hypothesis complexity},
  author={Liu, Tongliang and Lugosi, G{\'a}bor and Neu, Gergely and Tao, Dacheng},
  journal={arXiv preprint arXiv:1702.08712},
  year={2017}
}

@article{bousquet2002stability,
  title={Stability and generalization},
  author={Bousquet, Olivier and Elisseeff, Andr{\'e}},
  journal={Journal of Machine Learning Research},
  volume={2},
  number={Mar},
  pages={499--526},
  year={2002}
}


@article{mei2016landscape,
  title={The landscape of empirical risk for non-convex losses},
  author={Mei, Song and Bai, Yu and Montanari, Andrea},
  journal={arXiv preprint arXiv:1607.06534},
  year={2016}
}

@article{goldstein2016phasemax,
  title={PhaseMax: Convex phase retrieval via basis pursuit},
  author={Goldstein, Tom and Studer, Christoph},
  journal={arXiv preprint arXiv:1610.07531},
  year={2016}
}

@article{jaganathan2015phase,
  title={Phase retrieval: An overview of recent developments},
  author={Jaganathan, Kishore and Eldar, Yonina C and Hassibi, Babak},
  journal={arXiv preprint arXiv:1510.07713},
  year={2015}
}

@article{qu2017convolutional,
  title={Convolutional Phase Retrieval via Gradient Descent},
  author={Qu Qing and Yuqian Zhang and Yonina Eldar and John Wright},
  journal={Neural Information Processing Systems},
  year={2017}
}

@inproceedings{zhao2015nonconvex,
  title={A nonconvex optimization framework for low rank matrix estimation},
  author={Zhao, Tuo and Wang, Zhaoran and Liu, Han},
  booktitle={NIPS},
  pages={559--567},
  year={2015}
}

@article{bresler2016collaborative,
  title={Collaborative filtering with low regret},
  author={Bresler, Guy and Shah, Devavrat and Voloch, Luis Filipe},
  journal={ACM SIGMETRICS Performance Evaluation Review},
  volume={44},
  number={1},
  pages={207--220},
  year={2016},
}

@article{cai2017fast,
  title={Fast Rank One Alternating Minimization Algorithm for Phase Retrieval},
  author={Cai, Jian-Feng and Liu, Haixia and Wang, Yang},
  journal={arXiv preprint arXiv:1708.08751},
  year={2017}
}

@inproceedings{borgs2017thy,
  title={Thy friend is my friend: Iterative collaborative filtering for sparse matrix estimation},
  author={Borgs, Christian and Chayes, Jennifer and Lee, Christina E and Shah, Devavrat},
  booktitle={Advances in Neural Information Processing Systems},
  pages={4715--4726},
  year={2017}
}

@article{hastie2015matrix,
  title={Matrix completion and low-rank {SVD} via fast alternating least squares.},
  author={Hastie, Trevor and Mazumder, Rahul and Lee, Jason D and Zadeh, Reza},
  journal={Journal of Machine Learning Research},
  volume={16},
  pages={3367--3402},
  year={2015}
}

@article{tu2016low,
  title={Low-rank solutions of linear matrix equations via procrustes flow},
  author={Tu, S. and Boczar, R. and Simchowitz, M. and Soltanolkotabi, M. and Recht, B.},
  journal={ICML},
  pages={964--973},
  year={2016},
}

@article{allen2017natasha,
  title={Natasha 2: Faster non-convex optimization than sgd},
  author={Allen-Zhu, Zeyuan},
  journal={arXiv preprint arXiv:1708.08694},
  year={2017}
}

@article{agarwal2016finding,
  title={Finding approximate local minima for nonconvex optimization in linear time},
  author={Agarwal, Naman and Allen-Zhu, Zeyuan and Bullins, Brian and Hazan, Elad and Ma, Tengyu},
  journal={arXiv preprint arXiv:1611.01146},
  year={2016}
}

@inproceedings{jain2015fast,
  title={Fast exact matrix completion with finite samples},
  author={Jain, Prateek and Netrapalli, Praneeth},
  booktitle={Conference on Learning Theory},
  pages={1007--1034},
  year={2015}
}

@article{chen2017robust,
  title={Robust Wirtinger flow for phase retrieval with arbitrary corruption},
  author={Chen, Jinghui and Wang, Lingxiao and Zhang, Xiao and Gu, Quanquan},
  journal={arXiv preprint arXiv:1704.06256},
  year={2017}
}

@article{gunasekar2017implicit,
  title={Implicit Regularization in Matrix Factorization},
  author={Gunasekar, Suriya and Woodworth, Blake and Bhojanapalli, Srinadh and Neyshabur, Behnam and Srebro, Nathan},
  journal={arXiv preprint arXiv:1705.09280},
  year={2017}
}


@article{zhang2017nonconvex,
  title={A nonconvex approach for phase retrieval: Reshaped wirtinger flow and incremental algorithms},
  author={Zhang, Huishuai and Zhou, Yi and Liang, Yingbin and Chi, Yuejie},
  journal={The Journal of Machine Learning Research},
  volume={18},
  number={1},
  pages={5164--5198},
  year={2017}
}

@article{chen2018asymmetry,
  title={Asymmetry helps: Eigenvalue and eigenvector analyses of asymmetrically perturbed low-rank matrices},
  author={Chen, Yuxin and Cheng, Chen and Fan, Jianqing},
  journal={arXiv preprint arXiv:1811.12804},
  year={2018}
}

@article{li2018approximate,
  title={Approximate support recovery of atomic line spectral estimation: A tale of resolution and precision},
  author={Li, Qiuwei and Tang, Gongguo},
  journal={Applied and Computational Harmonic Analysis},
  year={2018},
  publisher={Elsevier}
}

@article{toh2010accelerated,
  title={An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems},
  author={Toh, Kim-Chuan and Yun, Sangwoon},
  journal={Pacific Journal of optimization},
  volume={6},
  number={615-640},
  pages={15},
  year={2010}
}

@article{beck2009fast,
  title={A fast iterative shrinkage-thresholding algorithm for linear inverse problems},
  author={Beck, Amir and Teboulle, Marc},
  journal={SIAM journal on imaging sciences},
  volume={2},
  number={1},
  pages={183--202},
  year={2009},
  publisher={SIAM}
}

@article{ma2011fixed,
  title={Fixed point and Bregman iterative methods for matrix rank minimization},
  author={Ma, Shiqian and Goldfarb, Donald and Chen, Lifeng},
  journal={Mathematical Programming},
  volume={128},
  number={1-2},
  pages={321--353},
  year={2011},
  publisher={Springer}
}

@inproceedings{shamir2011collaborative,
  title={Collaborative filtering with the trace norm: Learning, bounding, and transducing},
  author={Shamir, Ohad and Shalev-Shwartz, Shai},
  booktitle={Conference on Learning Theory},
  pages={661--678},
  year={2011}
}

@article{wang2016unified,
  title={A unified computational and statistical framework for nonconvex low-rank matrix estimation},
  author={Wang, Lingxiao and Zhang, Xiao and Gu, Quanquan},
  journal={arXiv preprint arXiv:1610.05275},
  year={2016}
}

@article{fornasier2011low,
  title={Low-rank matrix recovery via iteratively reweighted least squares minimization},
  author={Fornasier, Massimo and Rauhut, Holger and Ward, Rachel},
  journal={SIAM Journal on Optimization},
  volume={21},
  number={4},
  pages={1614--1640},
  year={2011},
  publisher={SIAM}
}

@article{vandereycken2013low,
  title={Low-rank matrix completion by Riemannian optimization},
  author={Vandereycken, Bart},
  journal={SIAM Journal on Optimization},
  volume={23},
  number={2},
  pages={1214--1236},
  year={2013},
  publisher={SIAM}
}

@article{barvinok1995problems,
  title={Problems of distance geometry and convex properties of quadratic maps},
  author={Barvinok, Alexander I.},
  journal={Discrete \& Computational Geometry},
  volume={13},
  number={2},
  pages={189--202},
  year={1995},
  publisher={Springer}
}

@article{cao2016poisson,
  title={Poisson matrix recovery and completion},
  author={Cao, Yang and Xie, Yao},
  journal={IEEE Transactions on Signal Processing},
  volume={64},
  number={6},
  pages={1609--1620},
  year={2016},
  publisher={IEEE}
}

@article{shapiro2018matrix,
  title={Matrix completion with deterministic pattern: A geometric perspective},
  author={Shapiro, Alexander and Xie, Yao and Zhang, Rui},
  journal={IEEE Transactions on Signal Processing},
  volume={67},
  number={4},
  pages={1088--1103},
  year={2019},
  publisher={IEEE}
}

@inproceedings{boumal2016non,
  title={The non-convex Burer-Monteiro approach works on smooth semidefinite programs},
  author={Boumal, Nicolas and Voroninski, Vlad and Bandeira, Afonso},
  booktitle={NIPS},
  pages={2757--2765},
  year={2016}
}

@article{lai2013improved,
  title={Improved iteratively reweighted least squares for unconstrained smoothed $\ell_q$ minimization},
  author={Lai, M. and Xu, Y. and Yin, W.},
  journal={SIAM Journal on Numerical Analysis},
  volume={51},
  number={2},
  pages={927--957},
  year={2013},
  publisher={SIAM}
}

@article{zhang2015accelerating,
  title={Accelerating parameter mapping with a locally low rank constraint},
  author={Zhang, Tao and Pauly, John M and Levesque, Ives R},
  journal={Magnetic resonance in medicine},
  volume={73},
  number={2},
  pages={655--661},
  year={2015},
  publisher={Wiley Online Library}
}


@article{yang2017misspecified,
  title={Misspecified Nonconvex Statistical Optimization for Phase Retrieval},
  author={Yang, Zhuoran and Yang, Lin F and Fang, Ethan X and Zhao, Tuo and Wang, Zhaoran and Neykov, Matey},
  journal={arXiv preprint arXiv:1712.06245},
  year={2017}
}


@article{wen2012solving,
  title={Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm},
  author={Wen, Zaiwen and Yin, Wotao and Zhang, Yin},
  journal={Mathematical Programming Computation},
  volume={4},
  number={4},
  pages={333--361},
  year={2012},
  publisher={Springer}
}


@inproceedings{sun2018markov,
  title={On {M}arkov chain gradient descent},
  author={Sun, Tao and Sun, Yuejiao and Yin, Wotao},
  booktitle={Advances in Neural Information Processing Systems},
  pages={9896--9905},
  year={2018}
}

@inproceedings{jain2010guaranteed,
  title={Guaranteed rank minimization via singular value projection},
  author={Jain, Prateek and Meka, Raghu and Dhillon, Inderjit S},
  booktitle={Advances in Neural Information Processing Systems},
  pages={937--945},
  year={2010}
}

@inproceedings{zhao2015nonconvex-estimation,
  title={A nonconvex optimization framework for low rank matrix estimation},
  author={Zhao, Tuo and Wang, Zhaoran and Liu, Han},
  booktitle={NIPS},
  pages={559--567},
  year={2015}
}

@article {MR1071714,
    AUTHOR = {Mathias, Roy},
     TITLE = {The spectral norm of a nonnegative matrix},
   JOURNAL = {Linear Algebra Appl.},
  FJOURNAL = {Linear Algebra and its Applications},
    VOLUME = {139},
      YEAR = {1990},
     PAGES = {269--284},
      ISSN = {0024-3795},
   MRCLASS = {15A60 (47A30)},
  MRNUMBER = {1071714},
MRREVIEWER = {Sing Cheong Ong},
       DOI = {10.1016/0024-3795(90)90403-Y},
       URL = {http://dx.doi.org/10.1016/0024-3795(90)90403-Y},
}

@inproceedings {schudy2012concentration,
    AUTHOR = {Schudy, Warren and Sviridenko, Maxim},
     TITLE = {Concentration and moment inequalities for polynomials of
              independent random variables},
 BOOKTITLE = {Proceedings of the {T}wenty-{T}hird {A}nnual {ACM}-{SIAM}
              {S}ymposium on {D}iscrete {A}lgorithms},
     PAGES = {437--446},
 PUBLISHER = {ACM, New York},
      YEAR = {2012},
   MRCLASS = {60E15 (05C80 60J27)},
  MRNUMBER = {3205227},
}

@article {dopico2000note,
    AUTHOR = {Dopico, Froil\'an M.},
     TITLE = {A note on {$\sin\Theta$} theorems for singular subspace
              variations},
   JOURNAL = {BIT},
  FJOURNAL = {BIT. Numerical Mathematics},
    VOLUME = {40},
      YEAR = {2000},
    NUMBER = {2},
     PAGES = {395--403},
      ISSN = {0006-3835},
   MRCLASS = {65F15 (15A18)},
  MRNUMBER = {1765743},
       DOI = {10.1023/A:1022303426500},
       URL = {http://dx.doi.org/10.1023/A:1022303426500},
}

@article {ChenCandes15solving,
    AUTHOR = {Chen, Yuxin and Cand\`es, Emmanuel J.},
     TITLE = {Solving random quadratic systems of equations is nearly as
              easy as solving linear systems},
   JOURNAL = {Comm. Pure Appl. Math.},
  FJOURNAL = {Communications on Pure and Applied Mathematics},
    VOLUME = {70},
      YEAR = {2017},
    NUMBER = {5},
     PAGES = {822--883},
      ISSN = {0010-3640},
   MRCLASS = {94A12 (90C90)},
  MRNUMBER = {3628877},
       DOI = {10.1002/cpa.21638},
       URL = {http://dx.doi.org/10.1002/cpa.21638},
}

@article {MR3371006,
    AUTHOR = {Yu, Y. and Wang, T. and Samworth, R. J.},
     TITLE = {A useful variant of the {D}avis-{K}ahan theorem for
              statisticians},
   JOURNAL = {Biometrika},
  FJOURNAL = {Biometrika},
    VOLUME = {102},
      YEAR = {2015},
    NUMBER = {2},
     PAGES = {315--323},
      ISSN = {0006-3444},
   MRCLASS = {15A42},
  MRNUMBER = {3371006},
       DOI = {10.1093/biomet/asv008},
       URL = {http://dx.doi.org/10.1093/biomet/asv008},
}

@article{ma2017inter,
  title={Inter-Subject Analysis: Inferring Sparse Interactions with Dense Intra-Graphs},
  author={Ma, Cong and Lu, Junwei and Liu, Han},
  journal={arXiv preprint arXiv:1709.07036},
  year={2017}
}

@article {MR0501589,
    AUTHOR = {ten Berge, Jos M. F.},
     TITLE = {Orthogonal {P}rocrustes rotation for two or more matrices},
   JOURNAL = {Psychometrika},
  FJOURNAL = {Psychometrika. A Journal Devoted to the Development of
              Psychology as a Quantitative Rational Science},
    VOLUME = {42},
      YEAR = {1977},
    NUMBER = {2},
     PAGES = {267--276},
      ISSN = {0033-3123},
   MRCLASS = {62H25 (62P15)},
  MRNUMBER = {0501589},
MRREVIEWER = {P. A. K. Covey-Crump},
       DOI = {10.1007/BF02294053},
       URL = {http://dx.doi.org/10.1007/BF02294053},
}

@article{li2018nonconvex,
  title={Nonconvex robust low-rank matrix recovery},
  author={Li, Xiao and Zhu, Zhihui and So, Anthony Man-Cho and Vidal, Rene},
  journal={arXiv preprint arXiv:1809.09237},
  year={2018}
}

@article{cai2018solving,
  title={Solving systems of phaseless equations via Riemannian optimization with optimal sampling complexity},
  author={Cai, Jian-Feng and Wei, Ke},
  journal={arXiv preprint arXiv:1809.02773},
  year={2018}
}

@article{cai2019accelerated,
  title={Accelerated alternating projections for robust principal component analysis},
  author={Cai, HanQin and Cai, Jian-Feng and Wei, Ke},
  journal={The Journal of Machine Learning Research},
  volume={20},
  number={1},
  pages={685--717},
  year={2019},
  publisher={JMLR. org}
}

@article{wei2016guarantees,
  title={Guarantees of Riemannian optimization for low rank matrix recovery},
  author={Wei, Ke and Cai, Jian-Feng and Chan, Tony F and Leung, Shingyu},
  journal={SIAM Journal on Matrix Analysis and Applications},
  volume={37},
  number={3},
  pages={1198--1222},
  year={2016},
  publisher={SIAM}
}

@article{chartrand2012nonconvex,
  title={Nonconvex splitting for regularized low-rank+ sparse decomposition},
  author={Chartrand, Rick},
  journal={IEEE Transactions on Signal Processing},
  volume={60},
  number={11},
  pages={5810--5819},
  year={2012},
  publisher={IEEE}
}

@article{zhang2011strongly,
  title={Strongly convex programming for exact matrix completion and robust principal component analysis},
  author={Zhang, Hui and Cai, Jian-Feng and Cheng, Lizhi and Zhu, Jubo},
  journal={arXiv preprint arXiv:1112.3946},
  year={2011}
}

@article{shen2014augmented,
  title={Augmented Lagrangian alternating direction method for matrix separation based on low-rank factorization},
  author={Shen, Yuan and Wen, Zaiwen and Zhang, Yin},
  journal={Optimization Methods and Software},
  volume={29},
  number={2},
  pages={239--263},
  year={2014},
  publisher={Taylor \& Francis}
}

@article{goldfarb2013fast,
  title={Fast alternating linearization methods for minimizing the sum of two convex functions},
  author={Goldfarb, Donald and Ma, Shiqian and Scheinberg, Katya},
  journal={Mathematical Programming},
  volume={141},
  number={1-2},
  pages={349--382},
  year={2013},
  publisher={Springer}
}

@article{chandrasekaran2012latent,
  title={Latent Variable Graphical Model Selection via Convex Optimization},
  author={Chandrasekaran, Venkat and Parrilo, Pablo A and Willsky, Alan S},
  journal={Annals of Statistics},
  volume={40},
  number={4},
  pages={1935--1967},
  year={2012},
  publisher={Institute of Mathematical Statistics}
}

@article{tao2011recovering,
  title={Recovering low-rank and sparse components of matrices from incomplete and noisy observations},
  author={Tao, Min and Yuan, Xiaoming},
  journal={SIAM Journal on Optimization},
  volume={21},
  number={1},
  pages={57--81},
  year={2011},
  publisher={SIAM}
}

@inproceedings{zhan2016online,
  title={Online (and offline) robust {PCA}: Novel algorithms and performance guarantees},
  author={Zhan, Jinchun and Lois, Brian and Guo, Han and Vaswani, Namrata},
  booktitle={Artificial intelligence and statistics},
  pages={1488--1496},
  year={2016}
}



@article{klopp2014noisy,
  title={Noisy low-rank matrix completion with general sampling distribution},
  author={Klopp, Olga},
  journal={Bernoulli},
  volume={20},
  number={1},
  pages={282--303},
  year={2014},
  publisher={Bernoulli Society for Mathematical Statistics and Probability}
}

@inproceedings{feng2013online,
  title={Online robust {PCA} via stochastic optimization},
  author={Feng, Jiashi and Xu, Huan and Yan, Shuicheng},
  booktitle={Advances in Neural Information Processing Systems},
  pages={404--412},
  year={2013}
}

@article {MR2906869,
    AUTHOR = {Koltchinskii, Vladimir and Lounici, Karim and Tsybakov,
              Alexandre B.},
     TITLE = {Nuclear-norm penalization and optimal rates for noisy low-rank
              matrix completion},
   JOURNAL = {Ann. Statist.},
  FJOURNAL = {The Annals of Statistics},
    VOLUME = {39},
      YEAR = {2011},
    NUMBER = {5},
     PAGES = {2302--2329},
      ISSN = {0090-5364},
   MRCLASS = {62J99 (15A83 60B20 62H12 62J07)},
  MRNUMBER = {2906869},
MRREVIEWER = {Goetz Trenkler},
       DOI = {10.1214/11-AOS894},
       URL = {http://dx.doi.org/10.1214/11-AOS894},
}

@article {tropp2012user,
    AUTHOR = {Tropp, Joel A.},
     TITLE = {User-friendly tail bounds for sums of random matrices},
   JOURNAL = {Found. Comput. Math.},
  FJOURNAL = {Foundations of Computational Mathematics. The Journal of the
              Society for the Foundations of Computational Mathematics},
    VOLUME = {12},
      YEAR = {2012},
    NUMBER = {4},
     PAGES = {389--434},
      ISSN = {1615-3375},
   MRCLASS = {60B20 (60F10 60G42 60G50)},
  MRNUMBER = {2946459},
       DOI = {10.1007/s10208-011-9099-z},
       URL = {http://dx.doi.org/10.1007/s10208-011-9099-z},
}

@inproceedings{johnson2013accelerating,
  title={Accelerating stochastic gradient descent using predictive variance reduction},
  author={Johnson, Rie and Zhang, Tong},
  booktitle={Advances in neural information processing systems},
  pages={315--323},
  year={2013}
}

@article {MR2994877,
    AUTHOR = {Hsu, Daniel and Kakade, Sham M. and Zhang, Tong},
     TITLE = {A tail inequality for quadratic forms of subgaussian random
              vectors},
   JOURNAL = {Electron. Commun. Probab.},
  FJOURNAL = {Electronic Communications in Probability},
    VOLUME = {17},
      YEAR = {2012},
     PAGES = {no. 52, 6},
      ISSN = {1083-589X},
   MRCLASS = {60E15 (60G42)},
  MRNUMBER = {2994877},
MRREVIEWER = {I. Cuculescu},
       DOI = {10.1214/ECP.v17-2079},
       URL = {http://dx.doi.org/10.1214/ECP.v17-2079},
}


@article {MR1176461,
    AUTHOR = {Schmitt, Bernhard A.},
     TITLE = {Perturbation bounds for matrix square roots and {P}ythagorean
              sums},
   JOURNAL = {Linear Algebra Appl.},
  FJOURNAL = {Linear Algebra and its Applications},
    VOLUME = {174},
      YEAR = {1992},
     PAGES = {215--227},
      ISSN = {0024-3795},
   MRCLASS = {15A60 (15A24 47A55 65F99)},
  MRNUMBER = {1176461},
MRREVIEWER = {Juan-Miguel Gracia},
       DOI = {10.1016/0024-3795(92)90052-C},
       URL = {http://dx.doi.org/10.1016/0024-3795(92)90052-C},
}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% citations from arxiv
@article{FWW17,
	title={Distributed Estimation of Principal Eigenspaces},
	author={Fan, Jianqing and Wang, Dong and Wang, Kaizheng and Zhu, Ziwei},
	journal={arXiv preprint arXiv:1702.06488},
	year={2017}
}

@article{rohde2011estimation,
  title={Estimation of high-dimensional low-rank matrices},
  author={Rohde, Angelika and Tsybakov, Alexandre B},
  journal={The Annals of Statistics},
  volume={39},
  number={2},
  pages={887--930},
  year={2011},
  publisher={Institute of Mathematical Statistics}
}

@article{wang2017solving2,
  title={Solving Almost all Systems of Random Quadratic Equations},
  author={Wang, Gang and Giannakis, Georgios B and Saad, Yousef and Chen, Jie},
  journal={arXiv preprint arXiv:1705.10407},
  year={2017}
}

@article{huang2017blind,
  title={Blind Deconvolution by a Steepest Descent Algorithm on a Quotient Manifold},
  author={Huang, Wen and Hand, Paul},
  journal={arXiv preprint arXiv:1710.03309},
  year={2017}
}



@article{rudelson2013hanson,
  title={{Hanson-Wright} inequality and sub-{G}aussian concentration},
  author={Rudelson, Mark and Vershynin, Roman},
  journal={Electronic Communications in Probability},
  volume={18},
  year={2013},
  publisher={The Institute of Mathematical Statistics and the Bernoulli Society}
}



@article{Soltanolkotabi17a,
  author    = {Mahdi Soltanolkotabi},
  title     = {Learning ReLUs via Gradient Descent},
  journal   = {arXiv preprint arXiv:1705.04591},
  year      = {2017},
  url       = {http://arxiv.org/abs/1705.04591},
  timestamp = {Wed, 07 Jun 2017 14:43:09 +0200},
  biburl    = {http://dblp.uni-trier.de/rec/bib/journals/corr/Soltanolkotabi17a},
  bibsource = {dblp computer science bibliography, http://dblp.org}
}



@article{Tropp:2015:IMC:2802188.2802189,
 author = {Tropp, Joel A.},
 title = {An Introduction to Matrix Concentration Inequalities},
 journal = {Found. Trends Mach. Learn.},
 issue_date = {05 2015},
 volume = {8},
 number = {1-2},
 month = may,
 year = {2015},
 issn = {1935-8237},
 pages = {1--230},
 numpages = {230},
 url = {http://dx.doi.org/10.1561/2200000048},
 doi = {10.1561/2200000048},
 acmid = {2802189},
 publisher = {Now Publishers Inc.},
 address = {Hanover, MA, USA},
}

@article{higham1992estimating,
  title={Estimating the matrix $p$-norm},
  author={Higham, Nicholas J},
  journal={Numerische Mathematik},
  volume={62},
  number={1},
  pages={539--555},
  year={1992},
  publisher={Springer}
}

@article{nesterov2006cubic,
  title={Cubic regularization of {N}ewton method and its global performance},
  author={Nesterov, Yurii and Polyak, Boris T},
  journal={Mathematical Programming},
  volume={108},
  number={1},
  pages={177--205},
  year={2006},
  publisher={Springer}
}

@article{chen2016projected,
  title={The projected power method: An efficient algorithm for joint alignment from pairwise differences},
  author={Chen, Yuxin and Cand\`es, Emmanuel},
  journal={Communications on Pure and Applied Mathematics},
  volume = {71},
  number = {8},
  pages = {1648-1714},
  year={2018}
}

@article{mathias1993perturbation,
  title={Perturbation bounds for the polar decomposition},
  author={Mathias, Roy},
  journal={SIAM Journal on Matrix Analysis and Applications},
  volume={14},
  number={2},
  pages={588--597},
  year={1993},
  publisher={SIAM}
}

@article{schniter2015compressive,
  title={Compressive phase retrieval via generalized approximate message passing},
  author={Schniter, Philip and Rangan, Sundeep},
  journal={IEEE Transactions on Signal Processing},
  volume={63},
  number={4},
  pages={1043--1055},
  year={2015},
  publisher={IEEE}
}


@inproceedings{zhang2016provable,
  title={Provable Non-convex Phase Retrieval with Outliers: Median Truncated {W}irtinger Flow},
  author={Zhang, Huishuai and Chi, Yuejie and Liang, Yingbin},
  booktitle={International conference on machine learning},
  pages={1022--1031},
  year={2016}
}


@inproceedings{ling2017fast,
  title={Fast blind deconvolution and blind demixing via nonconvex optimization},
  author={Ling, Shuyang and Strohmer, Thomas},
  booktitle={Sampling Theory and Applications (SampTA), 2017 International Conference on},
  pages={114--118},
  year={2017},
  organization={IEEE}
}

@article{wang2017solving,
  title={Solving systems of random quadratic equations via truncated amplitude flow},
  author={Wang, Gang and Giannakis, Georgios B and Eldar, Yonina C},
  journal={IEEE Transactions on Information Theory},
  year={2017},
  publisher={IEEE}
}

@article{zhong2017recovery,
  title={Recovery Guarantees for One-hidden-layer Neural Networks},
  author={Zhong, Kai and Song, Zhao and Jain, Prateek and Bartlett, Peter L and Dhillon, Inderjit S},
  journal={arXiv preprint arXiv:1706.03175},
  year={2017}
}



@article{dasarathy2013sketching,
  title={Sketching Sparse Matrices},
  author={Dasarathy, G. and Shah, P. and Bhaskar, B. N. and Nowak, R.},
  journal={arXiv preprint arXiv:1303.6544},
  year={2013}
}

@article{chen2014robust,
  title={Robust Spectral Compressed Sensing via Structured Matrix Completion},
  author={Chen, Yuxin and Chi, Yuejie},
  journal={IEEE Transactions on Information Theory},
  volume={60},
  number={10},
  year={2014},
  pages = {6576 - 6601}
}


@article{wedin1972perturbation,
  title={Perturbation bounds in connection with singular value decomposition},
  author={Wedin, Per-{\AA}ke},
  journal={BIT Numerical Mathematics},
  volume={12},
  number={1},
  pages={99--111},
  year={1972},
  publisher={Springer}
}
@article{lang1993real,
  title={Real and functional analysis},
  author={Lang, Serge},
  journal={Springer-Verlag, New York,},
  volume={10},
  pages={11--13},
  year={1993}
}

@article{DBLP:journals/corr/LiLSW16,
  author    = {Xiaodong Li and
               Shuyang Ling and
               Thomas Strohmer and
               Ke Wei},
  title     = {Rapid, Robust, and Reliable Blind Deconvolution via Nonconvex Optimization},
  journal   = {CoRR},
  volume    = {abs/1606.04933},
  year      = {2016},
  url       = {http://arxiv.org/abs/1606.04933},
  timestamp = {Wed, 07 Jun 2017 14:40:23 +0200},
  biburl    = {http://dblp.uni-trier.de/rec/bib/journals/corr/LiLSW16},
  bibsource = {dblp computer science bibliography, http://dblp.org}
}



@article{pananjady2019value,
  title={Instance-Dependent $\ell_{\infty}$-Bounds for Policy Evaluation in Tabular Reinforcement Learning},
  author={Pananjady, Ashwin and Wainwright, Martin J},
  journal={IEEE Transactions on Information Theory},
  volume={67},
  number={1},
  pages={566--585},
  year={2020},
  publisher={IEEE}
}


@article{martin2017worstranking,
  title={Worst-case vs Average-case Design for Estimation from Fixed Pairwise Comparisons},
  author={Ashwin Pananjady, Cheng Mao, Vidya Muthukumar, Martin J. Wainwright, Thomas A. Courtade},
  journal={arXiv preprint arXiv:1707.06217},
  year={2017}
}

@article{abbe2017entrywise,
  title={Entrywise Eigenvector Analysis of Random Matrices with Low Expected Rank},
  author={Abbe, Emmanuel and Fan, Jianqing and Wang, Kaizheng and Zhong, Yiqiao},
  journal={arXiv:1709.09565},
  year={2017}
}

@article{chen2015fast,
  title={Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees},
  author={Chen, Yudong and Wainwright, Martin J},
  journal={arXiv:1509.03025},
  year={2015}
}

@article{lee2010admira,
  title={Admira: Atomic decomposition for minimum rank approximation},
  author={Lee, Kiryung and Bresler, Yoram},
  journal={IEEE Transactions on Information Theory},
  volume={56},
  number={9},
  pages={4402--4416},
  year={2010},
  publisher={IEEE}
}

@article{mondelli2017fundamental,
  title={Fundamental Limits of Weak Recovery with Applications to Phase Retrieval},
  author={Mondelli, Marco and Montanari, Andrea},
  journal={arXiv preprint arXiv:1708.05932},
  year={2017}
}

@article{li2016nonconvex,
  title={The nonconvex geometry of low-rank matrix optimizations with general objective functions},
  author={Li, Qiuwei and Tang, Gongguo},
  journal={arXiv preprint arXiv:1611.03060},
  year={2016}
}

@article{tan2017phase,
  title={Phase Retrieval via Randomized Kaczmarz: Theoretical Guarantees},
  author={Tan, Yan Shuo and Vershynin, Roman},
  journal={arXiv preprint arXiv:1706.09993},
  year={2017}
}

@article{gao2016phase,
  title={Phase retrieval using {Gauss-Newton} method},
  author={Gao, Bing and Xu, Zhiqiang},
  journal={arXiv preprint arXiv:1606.08135},
  year={2016}
}

@article{chen2017probability,
  title={Regularized Mirror Descent: A Nonconvex Approach for Learning Mixed Probability Distributions},
  author={Yuxin Chen},
  year={2017}
}

@article{carpentier2015signal,
  title={On signal detection and confidence sets for low rank inference problems},
  author={Carpentier, Alexandra and Nickl, Richard},
  journal={Electronic Journal of Statistics},
  volume={9},
  number={2},
  pages={2675--2688},
  year={2015},
}

@article{duchi2017stochastic,
  title={Stochastic methods for composite optimization problems},
  author={Duchi, John and Ruan, Feng},
  journal={arXiv preprint arXiv:1703.08570},
  year={2017}
}

@article{duchi2017solving,
  title={Solving (most) of a set of quadratic equalities: Composite optimization for robust phase retrieval},
  author={Duchi, John C and Ruan, Feng},
  journal={arXiv:1705.02356, Information and Inference},
  year={2017}
}

@article{li2016symmetry,
  title={Symmetry, saddle points, and global geometry of nonconvex matrix factorization},
  author={Li, Xingguo and Wang, Zhaoran and Lu, Junwei and Arora, Raman and Haupt, Jarvis and Liu, Han and Zhao, Tuo},
  journal={arXiv preprint arXiv:1612.09296},
  year={2016}
}

@article{chen2015phase,
  title={Phase Retrieval with One or Two Diffraction Patterns by Alternating Projections with the Null Initialization},
  author={Chen, Pengwen and Fannjiang, Albert and Liu, Gi-Ren},
  journal={Journal of Fourier Analysis and Applications},
  pages={1--40},
  year={2015},
  publisher={Springer}
}

@article{chi2016kaczmarz,
  title={Kaczmarz method for solving quadratic equations},
  author={Chi, Yuejie and Lu, Yue M},
  journal={IEEE Signal Processing Letters},
  volume={23},
  number={9},
  pages={1183--1187},
  year={2016},
  publisher={IEEE}
}

@article{neyshabur2017geometry,
  title={Geometry of Optimization and Implicit Regularization in Deep Learning},
  author={Neyshabur, Behnam and Tomioka, Ryota and Salakhutdinov, Ruslan and Srebro, Nathan},
  journal={arXiv preprint arXiv:1705.03071},
  year={2017}
}

@inproceedings{lin2016generalization,
  title={Generalization properties and implicit regularization for multiple passes {SGM}},
  author={Lin, Junhong and Camoriano, Raffaello and Rosasco, Lorenzo},
  booktitle={International Conference on Machine Learning},
  pages={2340--2348},
  year={2016}
}

@article{zhang2016understanding,
  title={Understanding deep learning requires rethinking generalization},
  author={Zhang, Chiyuan and Bengio, Samy and Hardt, Moritz and Recht, Benjamin and Vinyals, Oriol},
  journal={arXiv preprint arXiv:1611.03530},
  year={2016}
}

@article{neyshabur2014search,
  title={In search of the real inductive bias: On the role of implicit regularization in deep learning},
  author={Neyshabur, Behnam and Tomioka, Ryota and Srebro, Nathan},
  journal={arXiv preprint arXiv:1412.6614},
  year={2014}
}


@article{wang2016sparse,
  title={Sparse phase retrieval via truncated amplitude flow},
  author={Wang, Gang and Zhang, Liang and Giannakis, Georgios B and Ak{\c{c}}akaya, Mehmet and Chen, Jie},
  journal={arXiv preprint arXiv:1611.07641},
  year={2016}
}

@article{chen2017spectral,
  title={Spectral Method and Regularized {MLE} Are Both Optimal for Top-{$K$} Ranking},
  author={Chen, Yuxin and Fan, Jianqing and Ma, Cong and Wang, Kaizheng},
  journal={Annals of Statistics},
  volume = {47},
  number = {4},
  pages = {2204-2235},
  month = {August},
  year={2019}
}


@article{bendory2017non,
  title={Non-convex phase retrieval from {STFT} measurements},
  author={Bendory, Tamir and Eldar, Yonina C and Boumal, Nicolas},
  journal={IEEE Transactions on Information Theory},
  year={2017},
  publisher={IEEE}
}

@article{zhong2017near,
  title={Near-optimal bound for phase synchronization},
  author={Zhong, Yiqiao and Boumal, Nicolas},
  journal={SIAM Journal on Optimization},
  year={2018}
}

@article{cai2016optimal,
  title={Optimal rates of convergence for noisy sparse phase retrieval via thresholded {W}irtinger flow},
  author={Cai, T Tony and Li, Xiaodong and Ma, Zongming},
  journal={The Annals of Statistics},
  volume={44},
  number={5},
  pages={2221--2251},
  year={2016},
  publisher={Institute of Mathematical Statistics}
}

@article{recht2011simpler,
  title={A simpler approach to matrix completion},
  author={Recht, Benjamin},
  journal={Journal of Machine Learning Research},
  volume={12},
  number={Dec},
  pages={3413--3430},
  year={2011}
}

@article{chen2015incoherence,
  title={Incoherence-optimal matrix completion},
  author={Chen, Yudong},
  journal={IEEE Transactions on Information Theory},
  volume={61},
  number={5},
  pages={2909--2923},
  year={2015},
  publisher={IEEE}
}

@article{davis1970rotation,
  title={The rotation of eigenvectors by a perturbation. III},
  author={Davis, Chandler and Kahan, William Morton},
  journal={{SIAM} Journal on Numerical Analysis},
  volume={7},
  number={1},
  pages={1--46},
  year={1970},
}

@article{ohlsson2011compressive,
  title={Compressive phase retrieval from squared output measurements via semidefinite programming},
  author={Ohlsson, H. and Yang, A. and Dong, R. and Sastry, S. S.},
  journal={Neural Information Processing Systems (NIPS)},
  year={2011}
}

@online{mahdiWFcode,
  key = 	 {what},
  author = 	 {E. J. Cand\`es and X. Li and M. Soltanolkotabi},
  title = 	 {\url{http://www-bcf.usc.edu/~soltanol/PRWF.html}},
  year = 	 {2014},
  OPTurl = 	 {http://www-bcf.usc.edu/~soltanol/PRWF.html},
  OPTannote = 	 {}
}


@article{repetti2014nonconvex,
  title={A nonconvex regularized approach for phase retrieval},
  author={Repetti, A. and Chouzenoux, E. and Pesquet, J-C},
  journal={International Conference on Image Processing},
  pages={1753--1757},
  year={2014},
}

@article{bahmani2015efficient,
  title={Efficient Compressive Phase Retrieval with Constrained Sensing Vectors},
  author={Bahmani, S. and Romberg, J.},
  journal={Advances in Neural Information Processing Systems},
  pages={523--531},
  year={2015}
}

@inproceedings{zheng2015convergent,
  title={A convergent gradient descent algorithm for rank minimization and semidefinite programming from random linear measurements},
  author={Zheng, Qinqing and Lafferty, John},
  booktitle={Advances in Neural Information Processing Systems},
  pages={109--117},
  year={2015}
}

@phdthesis{soltanolkotabi2014algorithms,
  title={ Algorithms and Theory for Clustering and Nonconvex Quadratic Programming},
  author={Soltanolkotabi, M.},
  year={2014},
  school={Stanford University}
}

@inproceedings{netrapalli2014non,
  title={Non-convex robust {PCA}},
  author={Netrapalli, P. and Niranjan, U. and Sanghavi, S. and Anandkumar, A. and Jain, P.},
  booktitle={Advances in Neural Information Processing Systems},
  pages={1107--1115},
  year={2014}
}

@article{white2015local,
  title={The local convexity of solving quadratic equations},
  author={White, C. D. and Ward, R. and Sanghavi, S.},
  journal={arXiv preprint arXiv:1506.07868},
  year={2015}
}

@article{supplement2015,
  title = {Supplemental Materials for: ``Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systems''},
  author = {Y. Chen and E. J. Cand\`es},
  month = {May},
  year = {2015}
}

@inproceedings{chen2014convex,
  title={A convex formulation for mixed regression with two components: Minimax optimal rates},
  author={Chen, Yudong and Yi, Xinyang and Caramanis, Constantine},
  booktitle={Conference on Learning Theory},
  pages={560--604},
  year={2014}
}



@article{repetti2014nonconvex,
  title={A nonconvex regularized approach for phase retrieval},
  author={Repetti, A. and Chouzenoux, E. and Pesquet, J.-C.},
  journal={International Conference on Image Processing (ICCP)},
  pages={1753--1757},
  year={2014},
}


@article{wei2015phase,
  title={Phase retrieval via {K}aczmarz methods},
  author={Wei, K.},
  journal={arXiv preprint arXiv:1502.01822},
  year={2015}
}


@article{elser2003phase,
  title={Phase retrieval by iterated projections},
  author={Elser, V.},
  journal={JOSA. A},
  volume={20},
  number={1},
  pages={40--55},
  year={2003},
}

@inproceedings{bhojanapalli2016global,
  title={Global optimality of local search for low rank matrix recovery},
  author={Bhojanapalli, Srinadh and Neyshabur, Behnam and Srebro, Nati},
  booktitle={Advances in Neural Information Processing Systems},
  pages={3873--3881},
  year={2016}
}


@article{tanner2016low,
  title={Low rank matrix completion by alternating steepest descent methods},
  author={Tanner, Jared and Wei, Ke},
  journal={Applied and Computational Harmonic Analysis},
  volume={40},
  number={2},
  pages={417--429},
  year={2016},
  publisher={Elsevier}
}

@article{burer2003nonlinear,
  title={A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization},
  author={Burer, Samuel and Monteiro, Renato DC},
  journal={Mathematical Programming},
  volume={95},
  number={2},
  pages={329--357},
  year={2003},
  publisher={Springer}
}



@inproceedings{park2017non,
  title={Non-square matrix sensing without spurious local minima via the Burer-Monteiro approach},
  author={Park, Dohyung and Kyrillidis, Anastasios and Carmanis, Constantine and Sanghavi, Sujay},
  booktitle={Artificial Intelligence and Statistics},
  pages={65--74},
  year={2017}
}

@article{zhao2015nonconvex,
  title={Nonconvex low rank matrix factorization via inexact first order oracle},
  author={Zhao, Tuo and Wang, Zhaoran and Liu, Han},
  journal={Advances in Neural Information Processing Systems},
  year={2015}
}



@article{yi2013alternating,
  title={Alternating minimization for mixed linear regression},
  author={Yi, X. and Caramanis, C. and Sanghavi, S.},
  journal={International Conference on Machine Learning},
  month = {June},
  year={2014}
}

@article{marchesini2007unified,
  title={A unified evaluation of iterative projection algorithms for phase retrieval},
  author={Marchesini, S.},
  year={2007},
  journal = {Review of Scientific Instruments},
  volume = {78},
  number = {1}
}

@article{hardt2013provable,
  title={Fast matrix completion without the condition number},
  author={Hardt, M. and Wootters, M.},
  journal={Conference on Learning Theory},
  year={2014},
  pages = {638 - 678}
}


@article{shechtman2014phase,
author={Shechtman, Y. and Eldar, Y. C. and Cohen, O. and Chapman, H.N. and J. Miao and Segev, M.},
journal={IEEE Signal Processing Magazine},
title={Phase Retrieval with Application to Optical Imaging},
year={2015},
month={May},
volume={32},
number={3},
pages={87-109},
keywords={image processing;reviews;biology;chemistry;engineering;optical imaging;phase retrieval;physics;review;signal processing;Extraterrestrial measurements;Holography;Optical diffraction;Optical imaging;Optical signal processing;Optical variables measurement;Phase measurement},
doi={10.1109/MSP.2014.2352673},
ISSN={1053-5888},}


@article{gross2014partial,
  title={A partial derandomization of PhaseLift using spherical designs},
  author={Gross, D. and Krahmer, F. and Kueng, R.},
  journal={Journal of Fourier Analysis and Applications},
  pages={229-266},
  volume = {21},
  number = {2},
  year={2015},
  publisher={Springer}
}

@article{bentkus2003inequality,
  title={An inequality for tail probabilities of martingales with differences bounded from one side},
  author={Bentkus, V},
  journal={Journal of Theoretical Probability},
  volume={16},
  number={1},
  pages={161--173},
  year={2003},
  publisher={Springer}
}

@book{ben2001lectures,
  title={Lectures on modern convex optimization},
  author={Ben-Tal, A. and Nemirovski, A.},
  volume={2},
  year={2001},
  publisher={{SIAM}}
}


@inproceedings{bandeira2014multireference,
  title={Multireference alignment using semidefinite programming},
  author={Bandeira, A. S. and Charikar, M. and Singer, A. and Zhu, A.},
  booktitle={Conference on Innovations in theoretical computer science},
  pages={459--470},
  year={2014},
}

@article{alexeev2014phase,
  title={Phase retrieval with polarization},
  author={Alexeev, B. and Bandeira, A. S. and Fickus, M. and Mixon, D. G.},
  journal={SIAM Journal on Imaging Sciences},
  volume={7},
  number={1},
  pages={35--66},
  year={2014},
  publisher={SIAM}
}


@article{huang2013consistent,
  title={Consistent shape maps via semidefinite programming},
  author={Huang, Q. and Guibas, L.},
  journal={Computer Graphics Forum},
  volume={32},
  number={5},
  pages={177--186},
  year={2013},
  organization={Wiley Online Library}
}

@article{chen2014matching,
  title={Near-Optimal Joint Optimal Matching via Convex Relaxation},
  author={Chen, Y. and L. J. Guibas and Q. Huang},
  journal = {International Conference on Machine Learning (ICML)},
  pages = {100 - 108},
  month = {June},
  year={2014}
}

@article{gross2014improved,
  title={Improved recovery guarantees for phase retrieval from coded diffraction patterns},
  author={Gross, D. and Krahmer, F. and Kueng, R.},
  journal={arXiv preprint arXiv:1402.6286},
  year={2014}
}

@article{arora2015simple,
  title={Simple, Efficient, and Neural Algorithms for Sparse Coding},
  author={Arora, S. and Ge, R. and Ma, T. and Moitra, A.},
  journal={arXiv:1503.00778},
  year={2015}
}

@book{nocedal2006numerical,
  title={Numerical Optimization (2nd edition)},
  author={Nocedal, J. and Wright, S. J.},
  year={2006},
  publisher={Springer}
}


@article{bubeck2015convex,
  title={Convex optimization: Algorithms and complexity},
  author={Bubeck, S{\'e}bastien},
  journal={Foundations and Trends{\textregistered} in Machine Learning},
  volume={8},
  number={3-4},
  pages={231--357},
  year={2015},
  publisher={Now Publishers, Inc.}
}


@book{trefethen1997numerical,
  title={Numerical linear algebra},
  author={Trefethen, L. N. and Bau III, D.},
  volume={50},
  year={1997},
  publisher={{SIAM}}
}

@article{sur2017likelihood,
  title={The Likelihood Ratio Test in High-Dimensional Logistic Regression Is Asymptotically a Rescaled $C$hi-Square},
  author={Sur, Pragya and Chen, Yuxin and Cand{\`e}s, Emmanuel J},
  journal={arXiv:1706.01191, accepted to Probability Theory and Related Fields},
  year={2017}
}

@article{zheng2016convergence,
  title={Convergence analysis for rectangular matrix completion using {B}urer-{M}onteiro factorization and gradient descent},
  author={Zheng, Qinqing and Lafferty, John},
  journal={arXiv:1605.07051},
  year={2016}
}

@article{el2015impact,
  title={On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators},
  author={El Karoui, Noureddine},
  journal={Probability Theory and Related Fields},
  pages={1--81},
  year={2015},
  publisher={Springer}
}

@article{javanmard2015biasing,
  title={De-biasing the lasso: Optimal sample size for {G}aussian designs},
  author={Javanmard, Adel and Montanari, Andrea},
  journal={arXiv preprint arXiv:1508.02757},
  year={2015}
}

@article{el2013robust,
  title={On robust regression with high-dimensional predictors},
  author={El Karoui, Noureddine and Bean, Derek and Bickel, Peter J and Lim, Chinghway and Yu, Bin},
  journal={Proceedings of the National Academy of Sciences},
  volume={110},
  number={36},
  pages={14557--14562},
  year={2013},
  publisher={National Acad Sciences}
}

@article{fienup1982phase,
  title={Phase retrieval algorithms: a comparison},
  author={Fienup, J. R.},
  journal={Applied optics},
  volume={21},
  pages={2758--2769},
  year={1982},
  publisher={Optical Society of America}
}

@article{ling2017regularized,
  title={Regularized Gradient Descent: A Nonconvex Recipe for Fast Joint Blind Deconvolution and Demixing},
  author={Ling, Shuyang and Strohmer, Thomas},
  journal={arXiv preprint arXiv:1703.08642},
  year={2017}
}

@article{balakrishnan2014statistical,
  title={Statistical guarantees for the {EM} algorithm: From population to sample-based analysis},
  author={Balakrishnan, S. and Wainwright, M. J. and Yu, B.},
  journal={arXiv preprint arXiv:1408.2156},
  year={2014}
}

@article{sun2016guaranteed,
  title={Guaranteed matrix completion via non-convex factorization},
  author={Sun, Ruoyu and Luo, Zhi-Quan},
  journal={IEEE Transactions on Information Theory},
  volume={62},
  number={11},
  pages={6535--6579},
  year={2016},
  publisher={IEEE}
}

@article{recht2010guaranteed,
  title={Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization},
  author={Recht, Benjamin and Fazel, Maryam and Parrilo, Pablo A},
  journal={SIAM review},
  volume={52},
  number={3},
  pages={471--501},
  year={2010},
  publisher={SIAM}
}

@book{cover2012elements,
  title={Elements of information theory},
  author={Cover, T. M. and Thomas, J. A.},
  year={2012},
  publisher={John Wiley \& Sons}
}

@book{tsybakov2009introduction,
  title={Introduction to nonparametric estimation},
  author={Tsybakov, A. B. and Zaiats, V.},
  volume={11},
  year={2009},
  publisher={Springer}
}

@book{mitzenmacher2005probability,
  title={Probability and computing},
  author={Mitzenmacher, M. and Upfal, E.},
  year={2005},
  publisher={Cambridge University Press}
}

@article{nielsen2013chi,
  title={On the {Chi} square and higher-order {Chi} distances for approximating $f$-divergences},
  author={Nielsen, Frank and Nock, Richard},
  journal={arXiv preprint arXiv:1309.3029},
  year={2013}
}

@article{candes2014wirtinger,
  title={Phase Retrieval via {Wirtinger} Flow: Theory and Algorithms},
  author={Cand\`es, E. and Li, X. and Soltanolkotabi, M.},
  journal={IEEE Transactions on Information Theory},
  volume = {61},
  number = {4},
  month = {April},
  year={2015},
  pages = {1985-2007}
}

@book{grenander1958toeplitz,
  title={Toeplitz forms and their applications},
  author={Grenander, Ulf and Szeg{\H{o}}, G{\'a}bor},
  year={1958},
  publisher={Univ of California Press}
}

@article{stewart1990matrix,
  title={Matrix perturbation theory},
  author={Stewart, Gilbert W and Sun, Ji-guang},
  year={1990},
  publisher={Academic press}
}

@article{gerchberg1972practical,
  title={A practical algorithm for the determination of phase from image and diffraction plane pictures},
  author={Gerchberg, R. W.},
  journal={Optik},
  volume={35},
  pages={237},
  year={1972}
}

@article{cai2015rop,
  title={{ROP}: Matrix recovery via rank-one projections},
  author={Cai, T. and Zhang, A.},
  journal={The Annals of Statistics},
  volume={43},
  number={1},
  pages={102--138},
  year={2015},
  publisher={Institute of Mathematical Statistics}
}



@inproceedings{liu2011universal,
  title={Universal low-rank matrix recovery from Pauli measurements},
  author={Liu, Yi-Kai},
  booktitle={Advances in Neural Information Processing Systems},
  pages={1638--1646},
  year={2011}
}

@article{tropp2010beyond,
  title={Beyond Nyquist: Efficient sampling of sparse bandlimited signals},
  author={Tropp, Joel A and Laska, Jason N and Duarte, Marco F and Romberg, Justin K and Baraniuk, Richard G},
  journal={Information Theory, IEEE Transactions on},
  volume={56},
  number={1},
  pages={520--544},
  year={2010},
  publisher={IEEE}
}
@article{aubrun2009almost,
  title={On almost randomizing channels with a short {K}raus decomposition},
  author={Aubrun, Guillaume},
  journal={Communications in Mathematical Physics},
  volume={288},
  number={3},
  pages={1103--1116},
  year={2009},
  publisher={Springer}
}

@article{talagrand1996majorizing,
  title={Majorizing measures: the generic chaining},
  author={Talagrand, Michel},
  journal={The Annals of Probability},
  pages={1049--1103},
  year={1996},
  publisher={JSTOR}
}

@inproceedings{johnstone2006high,
  title={High dimensional statistical inference and random matrices},
  author={Johnstone, Iain M},
  booktitle={Proceedings of the International Congress of Mathematicians: Madrid, August 22-30, 2006: invited lectures},
  pages={307--333},
  year={2006}
}

@inproceedings{liu2011universal,
  title={Universal low-rank matrix recovery from Pauli measurements},
  author={Liu, Yi-Kai},
  booktitle={Advances in Neural Information Processing Systems},
  pages={1638--1646},
  year={2011}
}


@article{waldspurger2012phase,
  title={Phase recovery, maxcut and complex semidefinite programming},
  author={Waldspurger, I. and d'Aspremont, A. and Mallat, S.},
  journal={Mathematical Programming},
  volume={149},
  number={1-2},
  pages={47--81},
  year={2015},
  publisher={Springer}
}

@book{ledoux1991probability,
  title={Probability in {B}anach Spaces: isoperimetry and processes},
  author={Ledoux, Michel and Talagrand, Michel},
  volume={23},
  year={1991},
  publisher={Springer}
}

@article{wei2015phase,
  title={Phase retrieval via {K}aczmarz methods},
  author={Wei, Ke},
  journal={arXiv preprint arXiv:1502.01822},
  year={2015}
}

@article{johnstone2001distribution,
  title={On the distribution of the largest eigenvalue in principal components analysis},
  author={Johnstone, I. M.},
  journal={The Annals of statistics},
  volume={29},
  number={2},
  pages={295--327},
  year={2001},
  publisher={Institute of Mathematical Statistics}
}

@article{kolte2016phase,
  title={Phase retrieval via incremental truncated {W}irtinger flow},
  author={Kolte, Ritesh and {\"O}zg{\"u}r, Ayfer},
  journal={arXiv preprint arXiv:1606.03196},
  year={2016}
}

@inproceedings{ge2016matrix,
  title={Matrix completion has no spurious local minimum},
  author={Ge, Rong and Lee, Jason D and Ma, Tengyu},
  booktitle={Advances in Neural Information Processing Systems},
  pages={2973--2981},
  year={2016}
}

@article{ge2017optimization,
  title={On the Optimization Landscape of Tensor Decompositions},
  author={Ge, Rong and Ma, Tengyu},
  journal={arXiv preprint arXiv:1706.05598},
  year={2017}
}

@article{sun2015complete,
  title={Complete dictionary recovery using nonconvex optimization},
  author={Sun, J. and Qu, Q. and Wright, J.},
  journal={International Conference on Machine Learning},
  pages={2351--2360},
  year={2015}
}

@article{cai2017spectral,
  title={Spectral Compressed Sensing via Projected Gradient Descent},
  author={Cai, Jian-Feng and Wang, Tianming and Wei, Ke},
  journal={arXiv preprint arXiv:1707.09726},
  year={2017}
}

@article{ge2017no,
  title={No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis},
  author={Ge, Rong and Jin, Chi and Zheng, Yi},
  journal={arXiv preprint arXiv:1704.00708},
  year={2017}
}

@article{sun2017complete,
  title={Complete dictionary recovery over the sphere I: Overview and the geometric picture},
  author={Sun, Ju and Qu, Qing and Wright, John},
  journal={IEEE Transactions on Information Theory},
  volume={63},
  number={2},
  pages={853--884},
  year={2017},
  publisher={IEEE}
}

@article{sun2015nonconvex,
  title={When Are Nonconvex Problems Not Scary?},
  author={Sun, J. and Qu, Q. and Wright, J.},
  journal={arXiv preprint arXiv:1510.06096},
  year={2015}
}


@article{kueng2017low,
  title={Low rank matrix recovery from rank one measurements},
  author={Kueng, Richard and Rauhut, Holger and Terstiege, Ulrich},
  journal={Applied and Computational Harmonic Analysis},
  volume={42},
  number={1},
  pages={88--116},
  year={2017},
  publisher={Elsevier}
}


@article{davis2007information,
  title={Information-theoretic metric learning},
  author={Davis, J. V. and Kulis, B. and Jain, P. and Sra, S. and Dhillon, I.S.},
  journal={International conference on Machine learning},
  pages={209--216},
  year={2007},
}

@article{chen2013exact,
  title={Exact and stable covariance estimation from quadratic sampling via convex programming},
  author={Chen, Y. and Chi, Y. and Goldsmith, A. J.},
  journal={IEEE Transactions on Information Theory},
  volume = {61},
  number = {7},
  pages = {4034--4059},
  year={2015}
}

@article{donoho2014minimax,
  title={Minimax risk of matrix denoising by singular value thresholding},
  author={Donoho, David and Gavish, Matan},
  journal={The Annals of Statistics},
  volume={42},
  number={6},
  pages={2413--2440},
  year={2014},
  publisher={Institute of Mathematical Statistics}
}

@ARTICLE{chi2012petrels,
author={Yuejie Chi and Eldar, Y.C. and Calderbank, R.},
journal={Signal Processing, IEEE Transactions on},
title={{PETRELS}: Parallel Subspace Estimation and Tracking by Recursive Least Squares From Partial Observations},
year={2013},
volume={61},
number={23},
pages={5947-5959},
keywords={computational complexity;data communication;direction-of-arrival estimation;least squares approximations;PETRELS;anomaly detection;computational complexity;high-dimensional manifold;internet traffic data;least-squares estimation;low-dimensional linear subspace;low-dimensional structure;low-dimensional subspace;low-rank matrix completion problem;network monitoring;parallel subspace estimation;partial observations;recursive least squares;recursive procedure;subspace matrix;tracking;video denoising;Approximation algorithms;Convergence;Estimation;Least squares approximations;Signal processing algorithms;Sparse matrices;Vectors;Matrix completion;online algorithms;partial observations;recursive least squares;subspace identification and tracking},
doi={10.1109/TSP.2013.2282910},
ISSN={1053-587X},}



@book{oja1983subspace,
  title={Subspace methods of pattern recognition},
  author={Oja, Erkki},
  volume={4},
  year={1983},
  publisher={Research Studies Press England}
}

@incollection{brand2002incremental,
  title={Incremental singular value decomposition of uncertain data with missing values},
  author={Brand, M.},
  booktitle={ECCV 2002},
  pages={707--720},
  year={2002},
  publisher={Springer}
}

@article{Balzano-2010,
title={Online Identification and Tracking of Subspaces from Highly Incomplete Information},
journal={Proc. Allerton 2010},
author={L. Balzano and R. Nowak and B. Recht},
year={2010}
}

@article{kulis2009low,
  title={Low-rank kernel learning with {B}regman matrix divergences},
  author={Kulis, B. and Sustik, M.A. and Dhillon, I.S.},
  journal={The Journal of Machine Learning Research},
  volume={10},
  pages={341--376},
  year={2009},
}

@article{marchesin2014alternating,
  title={Alternating Projection, Ptychographic Imaging and Phase Synchronization},
  author={Marchesin, S. and Tu, Y. and Wu, H.},
  journal={arXiv:1402.0550},
  year={2014}
}


@article{netrapalli2013phase,
  title={Phase Retrieval using Alternating Minimization},
  author={Netrapalli, P. and Jain, P. and Sanghavi, S.},
  journal={Advances in Neural Information Processing Systems (NIPS)},
  year={2013}
}



@article{Xu2013onlinePCA,
  title={Online {PCA} for Contaminated Data},
  author={J. Feng and H. Xu and S. Mannor and S. Yan},
  journal={Advances in Neural Information Processing Systems (NIPS)},
  year={2013},
}

@article{Caramanis2013onlinePCA,
  title={Memory Limited, Streaming {PCA}},
  author={I. Mitliagkas and C. Caramanis and P. Jain},
  journal={Advances in Neural Information Processing Systems (NIPS)},
  year={2013},
}



@article{gore2002mimo,
  title={{MIMO} antenna subset selection with space-time coding},
  author={Gore, D. A. and Paulraj, A. J.},
  journal={IEEE Transactions on Signal Processing},
  volume={50},
  number={10},
  pages={2580--2588},
  year={2002},
  publisher={IEEE}
}

@article{tian2012cyclic,
  title={Cyclic feature detection with sub-Nyquist sampling for wideband spectrum sensing},
  author={Tian, Z. and Tafesse, Y. and Sadler, B. M.},
  journal={IEEE Journal of Selected Topics in Signal Processing},
  volume={6},
  number={1},
  pages={58--69},
  year={2012},
  publisher={IEEE}
}


@inproceedings{jaganathan2012recovery,
  title={Recovery of sparse 1-{D} signals from the magnitudes of their {F}ourier transform},
  author={Jaganathan, K. and Oymak, S. and Hassibi, B.},
  booktitle={IEEE ISIT},
  pages={1473--1477},
  year={2012},
}

@article{balan2006signal,
  title={On signal reconstruction without phase},
  author={Balan, R. and Casazza, P. and Edidin, D.},
  journal={Applied and Computational Harmonic Analysis},
  volume={20},
  number={3},
  pages={345--356},
  year={2006},
  publisher={Elsevier}
}

@article{alexeev2012phase,
  title={Phase retrieval with polarization},
  author={Alexeev, B. and Bandeira, A. S. and Fickus, M. and Mixon, D. G.},
  journal={arXiv preprint arXiv:1210.7752},
  year={2012}
}

@inproceedings{tian2012compressed,
  title={Compressed sensing for MIMO radar: A stochastic perspective},
  author={Tian, Z. and Blasch, E.},
  booktitle={Statistical Signal Processing Workshop (SSP), 2012 IEEE},
  pages={548--551},
  year={2012},
  organization={IEEE}
}

@INPROCEEDINGS{Leus2011,
author={Leus, G. and Z. Tian},
booktitle={Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP),},
title={Recovering second-order statistics from compressive measurements},
year={2011},
pages={337-340},
keywords={compressed sensing;higher order statistics;random processes;signal detection;signal reconstruction;cognitive radio;compressive measurements;compressive sensing framework;cyclic power spectrum;general cyclostationary signals;least square reconstruction methods;random linear projections;second-order statistics;signal reconstruction;sparsity constraints;Cognitive radio;Compressed sensing;Correlation;Random processes;Sensors;Vectors;Wideband},
doi={10.1109/CAMSAP.2011.6136019},}



@ARTICLE{Hua1992,
author={Hua, Y.},
journal={IEEE Transactions on Signal Processing}, title={Estimating two-dimensional frequencies by matrix enhancement and matrix pencil},
year={1992},
month={Sep},
volume={40},
number={9},
pages={2267 -2280}, }



@book{muthukrishnan2005data,
  title={Data streams: Algorithms and applications},
  author={Muthukrishnan, S.},
  year={2005},
  publisher={Now Publishers Inc}
}


@article{davis2017nonsmooth,
  title={The nonsmooth landscape of phase retrieval},
  author={Davis, Damek and Drusvyatskiy, Dmitriy and Paquette, Courtney},
  journal={arXiv preprint arXiv:1711.03247},
  year={2017}
}



@article{jung2013compressive,
  title={Compressive Spectral Estimation for Nonstationary Random Processes},
  author={Jung, A. and Taubock, G. and Hlawatsch, F/},
  year={2013},
  journal={IEEE Transactions on Information Theory},
  volume = {59},
  number = {5},
  month = {May}
}


@article{gallager1968information,
  title={Information theory and reliable communication},
  author={Gallager, R. G.},
  year={1968},
  publisher={Wiley}
}

@article{raymer1994complex,
  title={Complex wave-field reconstruction using phase-space tomography},
  author={Raymer, M.G. and Beck, M. and McAlister, D.},
  journal={Physical review letters},
  volume={72},
  number={8},
  pages={1137},
  year={1994},
  publisher={APS}
}

@article{tian2012experimental,
  title={Experimental compressive phase space tomography},
  author={Tian, L. and Lee, J. and Oh, S. B. and Barbastathis, G.},
  journal={Optics Express},
  volume={20},
  number={8},
  pages={8296},
  year={2012},
  publisher={Optical Society of America}
}

@article{karoui2008operator,
  title={Operator norm consistent estimation of large-dimensional sparse covariance matrices},
  author={Karoui, N.},
  journal={The Annals of Statistics},
  pages={2717--2756},
  year={2008},
  publisher={JSTOR}
}

@article{baraniuk2008simple,
  title={A simple proof of the restricted isometry property for random matrices},
  author={Baraniuk, R. and Davenport, M. and DeVore, R. and Wakin, M.},
  journal={Constructive Approximation},
  volume={28},
  number={3},
  pages={253--263},
  year={2008},
  publisher={Springer}
}

@article{friedman2008sparse,
  title={Sparse inverse covariance estimation with the graphical lasso},
  author={Friedman, J. and Hastie, T. and Tibshirani, R.},
  journal={Biostatistics},
  volume={9},
  number={3},
  pages={432--441},
  year={2008},
  publisher={Biometrika Trust}
}

@inproceedings{fazel2004rank,
  title={Rank minimization and applications in system theory},
  author={Fazel, M and Hindi, H and Boyd, S},
  booktitle={American Control Conference},
  volume={4},
  pages={3273--3278},
  year={2004},
}

@article{daniels200760,
  title={60 {GHz} wireless communications: emerging requirements and design recommendations},
  author={Daniels, R. C. and Heath, R. W.},
  journal={IEEE Vehicular Technology Magazine},
  volume={2},
  number={3},
  pages={41--50},
  year={2007},
}

@book{golub2012matrix,
  title={Matrix computations},
  author={Golub, G. H. and van Loan, C. F.},
  volume={3},
  year={2012},
  publisher={JHU Press}
}


@article{negahban2011estimation,
  title={Estimation of (near) low-rank matrices with noise and high-dimensional scaling},
  author={Negahban, S. and Wainwright, M. J.},
  journal={The Annals of Statistics},
  volume={39},
  number={2},
  pages={1069--1097},
  year={2011},
  publisher={Institute of Mathematical Statistics}
}

@inproceedings{gore2000selecting,
  title={Selecting an optimal set of transmit antennas for a low rank matrix channel},
  author={Gore, D. A. and Nabar, R. U. and Paulraj, A.},
  booktitle={IEEE ICASSP},
  volume={5},
  pages={2785--2788},
  year={2000},
}


@book{scharf1991statistical,
  title={Statistical signal processing},
  author={Scharf, L. L.},
  volume={98},
  year={1991},
  publisher={Addison-Wesley Reading}
}

@article{schafer2005shrinkage,
  title={A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics},
  author={Sch{\"a}fer, J. and Strimmer, K.},
  journal={Statistical applications in genetics and molecular biology},
  volume={4},
  number={1},
  pages={32},
  year={2005}
}

@article{xu2020reanalysis,
  title={Reanalysis of Variance Reduced Temporal Difference Learning},
  author={Xu, Tengyu and Wang, Zhe and Zhou, Yi and Liang, Yingbin},
  journal={ICLR, arXiv preprint arXiv:2001.01898},
  year={2020}
}

@article{d2008first,
  title={First-order methods for sparse covariance selection},
  author={d'Aspremont, A. and Banerjee, O. and El Ghaoui, L.},
  journal={SIAM Journal on Matrix Analysis and Applications},
  volume={30},
  number={1},
  pages={56--66},
  year={2008},
  publisher={SIAM}
}

@article{bickel2008covariance,
  title={Covariance regularization by thresholding},
  author={Bickel, P. J. and Levina, E.},
  journal={The Annals of Statistics},
  volume={36},
  number={6},
  pages={2577--2604},
  year={2008},
  publisher={Institute of Mathematical Statistics}
}

@article{cai2011adaptive,
  title={Adaptive thresholding for sparse covariance matrix estimation},
  author={Cai, T. and Liu, W.},
  journal={Journal of the American Statistical Association},
  volume={106},
  number={494},
  year={2011}
}

@book{Tao2012RMT,
author={Terence Tao},
title={Topics in Random Matrix Theory},
year={2012},
address = {Providence, Rhode Island},
series = {Graduate Studies in Mathematics},
publisher = {American Mathematical Society}
}

@article{hanson1971bound,
  title={A bound on tail probabilities for quadratic forms in independent random variables},
  author={Hanson, D. L. and Wright, F. T.},
  journal={The Annals of Mathematical Statistics},
  volume={42},
  number={3},
  pages={1079--1083},
  year={1971},
  publisher={JSTOR}
}

@article{scharf1994matched,
  title={Matched subspace detectors},
  author={Scharf, L. L. and Friedlander, B.},
  journal={IEEE Transactions on Signal Processing},
  volume={42},
  number={8},
  pages={2146--2157},
  year={1994},
  publisher={IEEE}
}

@article{rudelson2013hanson,
  title={{H}anson-{W}right inequality and sub-{G}aussian concentration},
  author={Rudelson, M. and Vershynin, R.},
  journal={arXiv preprint arXiv:1306.2872},
  month = {June},
  year={2013}
}


@article{rudelson2008sparse,
  title={On sparse reconstruction from {Fourier and Gaussian} measurements},
  author={Rudelson, M. and Vershynin, R.},
  journal={Communications on Pure and Applied Mathematics},
  volume={61},
  number={8},
  pages={1025--1045},
  year={2008},
  publisher={Wiley Online Library}
}

@article{hsu2012tail,
  title={A tail inequality for quadratic forms of sub-{G}aussian random vectors},
  author={Hsu, D. and Kakade, S. M. and Zhang, T.},
  journal={Electronic Communications in Probability},
  year={2012},
  number = {52},
  volume = {17}
}

@phdthesis{kakade2003sample,
  title={On the sample complexity of reinforcement learning},
  author={Kakade, SM},
  year={2003},
  school={University of London}
}

@article{fienup1982phase,
  title={Phase retrieval algorithms: a comparison.},
  author={Fienup, J. R.},
  journal={Applied optics},
  volume={21},
  number={15},
  pages={2758--2769},
  year={1982}
}

@article{fienup1978reconstruction,
  title={Reconstruction of an object from the modulus of its {F}ourier transform},
  author={Fienup, J. R.},
  journal={Optics letters},
  volume={3},
  number={1},
  pages={27--29},
  year={1978},
  publisher={Optical Society of America}
}

@article{chan2008terahertz,
  title={Terahertz imaging with compressed sensing and phase retrieval},
  author={Chan, W. L. and Moravec, M. L. and Baraniuk, R. G. and Mittleman, D. M.},
  journal={Optics letters},
  volume={33},
  number={9},
  pages={974--976},
  year={2008},
  publisher={Optical Society of America}
}

@article{ohlsson2011compressive,
  title={Compressive phase retrieval from squared output measurements via semidefinite programming},
  author={Ohlsson, Henrik and Yang, Allen Y and Dong, Roy and Sastry, S Shankar},
  journal={arXiv preprint arXiv:1111.6323},
  year={2011}
}

@article{chen2017memory,
  title={Memory-efficient Kernel {PCA} via Partial Matrix Sampling and Nonconvex Optimization: a Model-free Analysis of Local Minima},
  author={Chen, Ji and Li, Xiaodong},
  journal={arXiv:1711.01742},
  year={2017}
}


@article{beck2012sparsity,
  title={Sparsity constrained nonlinear optimization: Optimality conditions and algorithms},
  author={Beck, A. and Eldar, Y. C.},
  journal={to appear in SIAM Optimization},
  year={2013}
}

@article{pfeiffer2006phase,
  title={Phase retrieval and differential phase-contrast imaging with low-brilliance X-ray sources},
  author={Pfeiffer, Franz and Weitkamp, Timm and Bunk, Oliver and David, Christian},
  journal={Nature physics},
  volume={2},
  number={4},
  pages={258--261},
  year={2006},
  publisher={Nature Publishing Group}
}

@article{coene1992phase,
  title={Phase retrieval through focus variation for ultra-resolution in field-emission transmission electron microscopy},
  author={Coene, Wim and Janssen, Guido and Op de Beeck, Marc and Van Dyck, Dirk},
  journal={Physical review letters},
  volume={69},
  number={26},
  pages={3743--3746},
  year={1992},
  publisher={APS}
}

@inproceedings{li2015phase,
  title={Phase retrieval using iterative projections: Dynamics in the large systems limit},
  author={Li, Gen and Gu, Yuantao and Lu, Yue M},
  booktitle={Allerton Conference on Communication, Control, and Computing},
  pages={1114--1118},
  year={2015},
  organization={IEEE}
}


@article{hao2018sparse,
  title={Sparse and Low-rank Tensor Estimation via Cubic Sketchings},
  author={Hao, Botao and Zhang, Anru and Cheng, Guang},
  journal={arXiv preprint arXiv:1801.09326},
  year={2018}
}

@book{kahane1993some,
  title={Some random series of functions},
  author={Kahane, Jean-Pierre},
  volume={5},
  year={1993},
  publisher={Cambridge University Press}
}

@article{demanet2012stable,
  title={Stable optimizationless recovery from phaseless linear measurements},
  author={Demanet, L. and Hand, P.},
  journal={Journal of Fourier Analysis and Applications},
  volume={20},
  number={1},
  pages={199--221},
  year={2014},
  publisher={Springer}
}

@article{Vershynin2012,
  title={Introduction to the non-asymptotic analysis of random matrices},
  author={Vershynin, R.},
  journal={Compressed Sensing, Theory and Applications},
  year={2012},
  pages = {210 - 268}
}

@article{candes2012phaselift,
  title={PhaseLift: Exact and stable signal recovery from magnitude measurements via convex programming},
  author={Cand\`es, Emmanuel and Strohmer, Thomas and Voroninski, Vladislav},
  journal={Communications on Pure and Applied Mathematics},
  pages = {1017--1026},
  volume = {66},
  number = {8},
  year={2013},
  publisher={Wiley Online Library}
}

@article{karlin1961moment,
  title={Moment inequalities of {P}olya frequency functions},
  author={Karlin, S. and Proschan, F. and Barlow, R. E.},
  journal={Pacific J. Math},
  volume={11},
  number={102},
  pages={3--1033},
  year={1961}
}

@article{candes2013phase,
  title={Phase retrieval via matrix completion},
  author={Cand\`es, E. J. and Eldar, Y. C. and Strohmer, T. and Voroninski, V.},
  journal={SIAM Journal on Imaging Sciences},
  volume={6},
  number={1},
  pages={199--225},
  year={2013},
  publisher={SIAM}
}

@article{balan2009painless,
  title={Painless reconstruction from magnitudes of frame coefficients},
  author={Balan, R. and Bodmann, B. and Casazza, P. and Edidin, D.},
  journal={Journal of Fourier Analysis and Applications},
  volume={15},
  number={4},
  pages={488--501},
  year={2009},
  publisher={Springer}
}

@article{candes2014phase,
  title={Phase retrieval from coded diffraction patterns},
  author={Cand\`es, E. J. and Li, X. and Soltanolkotabi, M.},
  journal={to appear in Applied and Computational Harmonic Analysis},
  year={2014},
}

@article{candes2012solving,
  title={Solving quadratic equations via {PhaseLift} when there are about as many equations as unknowns},
  author={Cand\`es, E. J. and Li, X.},
  journal={Foundations of Computational Mathematics},
  volume = {14},
  number = {5},
  pages = {1017-1026},
  year={2014}
}

@article{li2012sparse,
  title={Sparse signal recovery from quadratic measurements via convex programming},
  author={Li, X. and Voroninski, V.},
  journal={SIAM Journal on Mathematical Analysis},
  year={2013}
}


@article{bobkov2011concentration,
  title={Concentration of the information in data with log-concave distributions},
  author={Bobkov, S. and Madiman, Mokshay},
  journal={The Annals of Probability},
  volume={39},
  number={4},
  pages={1528--1543},
  year={2011},
  publisher={Institute of Mathematical Statistics}
}

@article{mazumder2010spectral,
  title={Spectral regularization algorithms for learning large incomplete matrices},
  author={Mazumder, R. and Hastie, T. and Tibshirani, R.},
  journal={Journal of machine learning research},
  volume={11},
  number={Aug},
  pages={2287--2322},
  year={2010}
}


@article{abernethy2009new,
  title={A new approach to collaborative filtering: Operator estimation with spectral regularization},
  author={Abernethy, Jacob and Bach, Francis and Evgeniou, Theodoros and Vert, Jean-Philippe},
  journal={The Journal of Machine Learning Research},
  volume={10},
  pages={803--826},
  year={2009},
}

@article{vetterli2002sampling,
  title={Sampling signals with finite rate of innovation},
  author={Vetterli, Martin and Marziliano, Pina and Blu, Thierry},
  journal={IEEE Transactions on Signal Processing},
  volume={50},
  number={6},
  pages={1417--1428},
  year={2002},
  publisher={IEEE}
}

@article{chi2013compressive,
  title={Compressive Demodulation of Mutually Interfering Signals},
  author={Chi, Yuejie and Xie, Yao and Calderbank, Robert},
  journal={submitted to IEEE Transactions on Information Theory},
  url = {http://arxiv.org/abs/1303.3904},
  year={2013}
}


@article{CandesPlan2011Tight,
author={Cand\`es, E.J. and Plan, Y.},
journal={IEEE Transactions on Information Theory}, title={Tight Oracle Inequalities for Low-Rank Matrix Recovery From a Minimal Number of Noisy Random Measurements},
year={2011},
volume={57},
number={4},
pages={2342-2359},
keywords={probability;singular value decomposition;sparse matrices;adaptive oracle error;constant number;linear combination;low-rank matrix recovery;matrix column space;matrix entry;noisy data;noisy random measurement;nuclear-norm minimization;recovery error;restricted isometry property;tight oracle inequality;Compressed sensing;Linear matrix inequalities;Measurement uncertainty;Minimization;Noise;Noise measurement;Sparse matrices;Convex optimization;Dantzig selector;matrix completion;norm of random matrices;oracle inequalities and semidefinite programming},
}


@article{LustigHankel2013,
  title={Calibrationless Parallel Imaging Reconstruction Based on Structured Low-Rank Matrix Completion},
  author={Shin, Peter J and Larson, Peder EZ and Ohliger, Michael A and Elad, Michael and Pauly, John M and Vigneron, Daniel B and Lustig, Michael},
  year = {2012},
  journal = {submitted to Magnetic Resonance in Medicine}
}

@book{Alon2008,
  title={The Probabilistic Method (3rd Edition)},
  author={Noga Alon and Joel H. Spencer},
  publisher={Wiley},
  year={2008},
}

@article{koren2009matrix,
  title={Matrix factorization techniques for recommender systems},
  author={Koren, Yehuda and Bell, Robert and Volinsky, Chris},
  journal={Computer},
  number={8},
  pages={30--37},
  year={2009},
  publisher={IEEE}
}


@article{zhang2013,
  title={Fast {3D DCE-MRI} with Sparsity and Low-Rank enhanced {SPIR}i{T} ({SLR-SPIRiT})},
  author={Zhang, T and Alley, MT and Lustig, M and Li, X and Pauly, J.M. and Vasanawala, SS},
  journal={Proceedings of the 21th Annual Meeting of ISMRM},
  pages = {2624},
  year={2013},
}

@article{wu2007collaborative,
  title={Collaborative filtering via ensembles of matrix factorizations},
  author={Wu, Mingrui},
  booktitle={Proceedings of KDD Cup and Workshop},
  volume={2007},
  year={2007}
}


@article{duarte2012spectral,
  title={Spectral compressive sensing},
  author={Duarte, M.F. and Baraniuk, R.G.},
  journal={Applied and Computational Harmonic Analysis},
  year={2012},
}

@article{tang2013minimax,
  title={Near Minimax Line Spectral Estimation},
  author={G. Tang and B. Bhaskar and B. Recht},
  url={http://arxiv.org/abs/1303.4348},
  year={2013},
}

@article{candes2008restricted,
  title={The restricted isometry property and its implications for compressed sensing},
  author={Cand{\`e}s, E. J.},
  journal={Comptes Rendus Mathematique},
  volume={346},
  number={9},
  pages={589--592},
  year={2008},
  publisher={Elsevier}
}

@article{CandesRombergTao2006Stable,
author = {Cand\`es, E. J. and Romberg, J. K. and Tao, T.},
title = {Stable signal recovery from incomplete and inaccurate measurements},
journal = {Communications on Pure and Applied Mathematics},
volume = {59},
number = {8},
publisher = {Wiley Subscription Services, Inc., A Wiley Company},
pages = {1207--1223},
year = {2006},
}

@ARTICLE{CandesPlan2011RIPless,
author={Cand\`es, E.J. and Plan, Y.},
journal={IEEE Transactions on Information Theory}, title={A Probabilistic and {RIP}less Theory of Compressed Sensing},
year={2011},
volume={57},
number={11},
pages={7235-7254},
keywords={Fourier analysis;data compression;random processes;signal reconstruction;statistical distributions;Fourier coefficients;Gaussian model;RIPless theory;compressed sensing;frequency measurements;probabilistic theory;probability distribution;restricted isometry property;signal random model;sparse signals;Coherence;Compressed sensing;Convolution;Discrete Fourier transforms;Upper bound;(weak) restricted isometries; $ell_1$ minimization;Compressed sensing;Dantzig selector;Gross' golfing scheme;LASSO;operator Bernstein inequalities;random matrices;sparse regression},
doi={10.1109/TIT.2011.2161794},
ISSN={0018-9448},}


@article{gedalyahu2011multichannel,
  title={Multichannel sampling of pulse streams at the rate of innovation},
  author={Gedalyahu, Kfir and Tur, Ronen and Eldar, Yonina. C.},
  journal={IEEE Transactions on Signal Processing},
  volume={59},
  number={4},
  pages={1491--1504},
  year={2011},
  publisher={IEEE}
}


@article{potter2010sparsity,
  title={Sparsity and compressed sensing in radar imaging},
  author={Potter, L.C. and Ertin, E. and Parker, J.T. and Cetin, M.},
  journal={Proceedings of the IEEE},
  volume={98},
  number={6},
  pages={1006--1020},
  year={2010},
  publisher={IEEE}
}

@article{schermelleh2010guide,
  title={A guide to super-resolution fluorescence microscopy},
  author={Schermelleh, L. and Heintzmann, R. and Leonhardt, H.},
  journal={The Journal of cell biology},
  volume={190},
  number={2},
  pages={165--175},
  year={2010},
}

@article{shechtman2011sparsity,
  title={Sparsity based sub-wavelength imaging with partially incoherent light via quadratic compressed sensing},
  author={Shechtman, Y. and Eldar, Y. C. and Szameit, A. and Segev, M.},
  journal={Optics express},
  volume={19},
  number={16},
  year={2011},
  publisher={Optical Society of America}
}


@article{li2011compressed,
year={2013},
journal={Constructive Approximation},
volume={37},
issue={1},
title={Compressed Sensing and Matrix Completion with Constant Proportion of Corruptions},
author={Li, Xiaodong},
pages={73-99},
language={English}
}

@article{li2013sparse,
  title={Sparse signal recovery from quadratic measurements via convex programming},
  author={Li, X. and Voroninski, V.},
  journal={SIAM Journal on Mathematical Analysis},
  volume={45},
  number={5},
  pages={3019--3033},
  year={2013},
  publisher={SIAM}
}

@article{cai2010singular,
  title={A singular value thresholding algorithm for matrix completion},
  author={Cai, J. F. and Cand\`es, E. J. and Shen, Z.},
  journal={SIAM Journal on Optimization},
  volume={20},
  number={4},
  pages={1956--1982},
  year={2010},
  publisher={SIAM}
}

@ARTICLE{HuaSarkar1990,
author={Hua, Y. and Sarkar, T. K.},
journal={IEEE Transactions on Acoustics, Speech and Signal Processing}, title={Matrix pencil method for estimating parameters of exponentially damped/undamped sinusoids in noise},
year={1990},
month={may},
volume={38},
number={5},
pages={814 -824},
}

@article{Huang2008three,
  title={Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy},
  author={Huang, B. and Wang, W. and Bates, M. and Zhuang, X.},
  journal={Science},
  volume={319},
  number={5864},
  pages={810--813},
  month = {February},
  year={2008},
  publisher={American Association for the Advancement of Science}
}

@article{AbaMenHar1991,
author={Abatzoglou, T. J. and Mendel, J. M. and Harada, G. A.},
journal={IEEE Transactions on Signal Processing}, title={The constrained total least squares technique and its applications to harmonic superresolution},
year={1991},
month={May},
volume={39},
number={5},
pages={1070 -1087},
}

@article{Borcea2002imaging,
  title={Imaging and time reversal in random media},
  author={Borcea, L. and Papanicolaou, G. and Tsogka, C. and Berryman, J.},
  journal={Inverse Problems},
  volume={18},
  number={5},
  pages={1247},
  year={2002},
}

@article{Sankaranarayanan2010compressive,
  title={Compressive acquisition of dynamic scenes},
  author={Sankaranarayanan, A. and Turaga, P. and Baraniuk, R. and Chellappa, R.},
  journal={Computer Vision--ECCV 2010},
  pages={129--142},
  year={2010},
}


@article{Balle2012spectral,
  title={Spectral learning of general weighted automata via constrained matrix completion},
  author={Balle, B. and Mohri, M.},
  journal={Advances in Neural Information Processing Systems (NIPS)},
  pages={2168--2176},
  year={2012}
}


@ARTICLE{DragVetterliBlu2007,
author={Dragotti, P. L. and Vetterli, M. and Blu, T.},
journal={IEEE Transactions on Signal Processing}, title={Sampling Moments and Reconstructing Signals of Finite Rate of Innovation: {S}hannon Meets Strang-Fix},
year={2007},
month={May},
volume={55},
number={5},
pages={1741 -1757},
}


@ARTICLE{VanMittra1978,
author={VanBlaricum, M. and Mittra, R.},
journal={IEEE Transactions on Antennas and Propagation}, title={Problems and solutions associated with {P}rony's method for processing transient data},
year={1978},
month={Jan},
volume={26},
number={1},
pages={ 174 - 182},
}


@ARTICLE{SayeedAazhang1999,
author={Sayeed, A. M. and Aazhang, B.},
journal={IEEE Transactions on Communications}, title={Joint multipath-{D}oppler diversity in mobile wireless communications},
year={1999},
month={Jan},
volume={47},
number={1},
pages={123 -132},
}


@ARTICLE{NionSid2010,
author={Nion, D. and Sidiropoulos, N. D.},
journal={IEEE Transactions on Signal Processing}, title={Tensor Algebra and Multidimensional Harmonic Retrieval in Signal Processing for {MIMO Radar}},
year={2010},
month={Nov. },
volume={58},
number={11},
pages={5693 -5705},
}


@ARTICLE{SwamiMendel1991,
author={Swami, A. and Mendel, J.M.},
journal={IEEE Transactions on Signal Processing}, title={Cumulant-based approach to harmonic retrieval and related problems},
year={1991},
month={May},
volume={39},
number={5},
pages={1099 -1109},
}


@misc{Fazel2011hankel,
  title={Hankel matrix rank minimization with applications in system identification and realization},
  author={Fazel, M. and Pong, T. K. and Sun, D. and Tseng, P.},
  year={2011},
  journal = {submitted}
}

@article{Markovsky2008structured,
  title={Structured low-rank approximation and its applications},
  author={Markovsky, I.},
  journal={Automatica},
  volume={44},
  number={4},
  pages={891--909},
  year={2008},
  publisher={Elsevier}
}

@article{Chu2003structured,
  title={Structured low rank approximation},
  author={Chu, M. T. and Funderlic, R. E. and Plemmons, R. J.},
  journal={Linear algebra and its applications},
  volume={366},
  pages={157--172},
  year={2003},
  publisher={Elsevier}
}

@phdthesis{fazel2002matrix,
  title={Matrix rank minimization with applications},
  author={Fazel, Maryam},
  year={2002}
}



@article{Fazel2003Hankel,
author={Fazel, M. and Hindi, H. and Boyd, S. P.},
journal={American Control Conference}, title={Log-det heuristic for matrix rank minimization with applications to {H}ankel and {E}uclidean distance matrices},
year={2003},
}

@article{lustig2007sparse,
  title={Sparse {MRI}: The application of compressed sensing for rapid {MR} imaging},
  author={Lustig, M. and Donoho, D. and Pauly, J. M.},
  journal={Magnetic Resonance in Medicine},
  volume={58},
  number={6},
  pages={1182--1195},
  year={2007},
}

@INPROCEEDINGS{UJSPC09,
  AUTHOR =       {Y. Chi and L. L. Scharf and A. Pezeshki and R. Calderbank},
  TITLE =        {Sensitivity to basis mismatch of compressed sensing for spectrum analysis and beamforming},
  BOOKTITLE =    {Proc. 6th U.S./Australia Joint Workshop on Defence Applications of Signal Processing (DASP)},
  YEAR =         {2009},
  address =      {Lihue, HI},
  month =        {Sep. 26-28},
}

@article{UJPSC09,
  AUTHOR =       {Y. Chi and A. Pezeshki and L. L. Scharf and R. Calderbank},
  TITLE =        {Sensitivity to basis mismatch in compressed sensing},
  journal =    {ICASSP},
  YEAR =         {2010},
  address =      {Dallas, TX},
  month =        {Mar.},
}


@article{PeledYes2001,
author = {Peled, S. and Yeshurun, Y.},
title = {Superresolution in {MRI}: Application to human white matter fiber tract visualization by diffusion tensor imaging},
journal = {Magnetic Resonance in Medicine},
volume = {45},
number = {1},
publisher = {John Wiley & Sons, Inc.},
issn = {1522-2594},
pages = {29--35},
keywords = {high-resolution MRI, echo-planar imaging, white matter tractography in vivo, brain conductivity mapping, single-shot imaging},
year = {2001},
}

@article{Odendaal1994two,
  title={Two-dimensional superresolution radar imaging using the {MUSIC} algorithm},
  author={Odendaal, J. W. and Barnard, E. and Pistorius, CWI},
  journal={IEEE Transactions on Antennas and Propagation},
  volume={42},
  number={10},
  month = {Oct},
  pages={1386--1391},
  year={1994},
  publisher={IEEE}
}


@article{Park2003,
author={Sung Cheol Park and Min Kyu Park and Moon Gi Kang},
journal={IEEE Signal Processing Magazine}, title={Super-resolution image reconstruction: a technical overview},
year={2003},
month={May},
volume={20},
number={3},
pages={ 21 - 36},
}


@article{FreemanJonesPasztor2002,
author={Freeman, W.T. and Jones, T.R. and Pasztor, E.C.},
journal={IEEE Computer Graphics and Applications}, title={Example-based super-resolution},
year={2002},
month={Mar/Apr},
volume={22},
number={2},
pages={56 -65},
}


@article{Bajwa2010compressed,
  title={Compressed channel sensing: A new approach to estimating sparse multipath channels},
  author={Bajwa, W.U. and Haupt, J. and Sayeed, A.M. and Nowak, R.},
  journal={Proceedings of the IEEE},
  volume={98},
  number={6},
  pages={1058--1076},
  year={2010},
  publisher={IEEE},
  month = {June}
}

@inproceedings{sun2016geometric,
  title={A geometric analysis of phase retrieval},
  author={Sun, Ju and Qu, Qing and Wright, John},
  booktitle={Information Theory (ISIT), 2016 IEEE International Symposium on},
  pages={2379--2383},
  year={2016},
  organization={IEEE}
}

@article{Chandrasekaran2010convex,
  title={The convex algebraic geometry of linear inverse problems},
  author={Chandrasekaran, V. and Recht, B. and Parrilo, P.A. and Willsky, A.S.},
  journal={48th Annual Allerton Conference on Communication, Control, and Computing},
  pages={699--703},
  year={2010},
  organization={IEEE}
}


@article{Chi2011sensitivity,
  title={Sensitivity to basis mismatch in compressed sensing},
  author={Chi, Y. and Scharf, L.L. and Pezeshki, A. and Calderbank, A.R.},
  journal={IEEE Transactions on Signal Processing},
  volume={59},
  number={5},
  month = {May},
  pages={2182--2195},
  year={2011},
  publisher={IEEE}
}


@article{Herman2009high,
  title={High-resolution radar via compressed sensing},
  author={Herman, M.A. and Strohmer, T.},
  journal={IEEE Transactions on Signal Processing},
  volume={57},
  number={6},
  pages={2275--2284},
  month = {June},
  year={2009},
  publisher={IEEE}
}


@article{Berger2010sparse,
  title={Sparse channel estimation for multicarrier underwater acoustic communication: From subspace methods to compressed sensing},
  author={Berger, C.R. and Zhou, S. and Preisig, J.C. and Willett, P.},
  journal={IEEE Transactions on Signal Processing},
  month = {March},
  volume={58},
  number={3},
  pages={1708--1721},
  year={2010},
}


@article{Schmidt1986MUSIC,
  title={Multiple emitter location and signal parameter estimation},
  author={Schmidt, R.},
  journal={IEEE Transactions on Antennas and Propagation},
  volume={34},
  number={3},
  pages={276--280},
  year={1986},
}


@ARTICLE{TuftsKumaresan1982,
author={Tufts, D.W. and Kumaresan, R.},
journal={Proceedings of the IEEE}, title={Estimation of frequencies of multiple sinusoids: Making linear prediction perform like maximum likelihood},
year={1982},
month={sept.},
volume={70},
number={9},
pages={ 975 - 989},
keywords={Apertures;Data mining;Eigenvalues and eigenfunctions;Filters;Frequency estimation;Gaussian noise;Least squares approximation;Matrix decomposition;Maximum likelihood estimation;Signal to noise ratio;},
doi={10.1109/PROC.1982.12428},
ISSN={0018-9219},}



@ARTICLE{RoyKailathESPIRIT1989,
author={Roy, R. and Kailath, T.},
journal={IEEE Transactions on Acoustics, Speech and Signal Processing}, title={{ESPRIT}-estimation of signal parameters via rotational invariance techniques},
year={1989},
month={Jul},
volume={37},
number={7},
pages={984 -995},
}


@article{Gross2011recovering,
  title={Recovering low-rank matrices from few coefficients in any basis},
  author={Gross, David},
  journal={IEEE Transactions on Information Theory},
  month = {March},
  volume={57},
  number={3},
  pages={1548--1566},
  year={2011},
}

@article{molybogno2019,
  title={No Spurious Solutions in Non-convex Matrix Sensing: Structure Compensates for Isometry},
  author={Molybog, Igor and Sojoudi, Somayeh and Lavaei, Javad},
  year = {2019}
}

@book{VonNeumannOrthogonalSpaces,
  title={Functional Operators, Volume 2: The Geometry of Orthogonal Spaces},
  author={J. von Neumann},
  year={1950},
  publisher={Princeton University Press}
}


@ARTICLE{CandesFernandez2012SR,
author={E. J. Cand\`es and C. Fernandez-Granda},
title={Towards a Mathematical Theory of Super-Resolution},
year={2013},
journal = {to appear in Communications on Pure and Applied Mathematics},}

@ARTICLE{CandesFernandez2012SRNoisy,
author={E. J. Cand\`es and C. Fernandez-Granda},
title={Super-Resolution from Noisy Data},
year={2012},
month={November},
url = {http://arxiv.org/abs/1211.0290},}

@ARTICLE{TangBhaskarShahRecht2012,
author={G. Tang and B. N. Bhaskar and P. Shah and B. Recht},
title={Compressed Sensing Off the Grid},
year={2012},
month={July},
url = {http://arxiv.org/abs/1207.6053},}


@article{zhang2018much,
  title={How much restricted isometry is needed in nonconvex matrix recovery?},
  author={Zhang, Richard and Josz, C{\'e}dric and Sojoudi, Somayeh and Lavaei, Javad},
  booktitle={NeurIPS},
  pages={5586--5597},
  year={2018}
}

@article{zhang2019sharp,
  title={Sharp Restricted Isometry Bounds for the Inexistence of Spurious Local Minima in Nonconvex Matrix Recovery},
  author={Zhang, R. and Sojoudi, S. and Lavaei, J.},
  journal={arXiv:1901.01631},
  year={2019}
}

@article{ExactMC09,
    author = {Cand\`es, Emmanuel and Recht, Benjamin},
    title = {Exact Matrix Completion via Convex Optimization},
    journal = {Foundations of Computational Mathematics},
    volume = {9},
    number = {6},
    pages = {717-772},
    month = {April},
    year = {2009},
}

@inproceedings{hardt2014understanding,
  title={Understanding alternating minimization for matrix completion},
  author={Hardt, Moritz},
  booktitle={Foundations of Computer Science (FOCS)},
  pages={651--660},
  year={2014},
}

@article{rennie2005fast,
  title={Fast maximum margin matrix factorization for collaborative prediction},
  author={Rennie, Jasson DM and Srebro, Nathan},
  journal={International conference on Machine learning},
  pages={713--719},
  year={2005},
}

@inproceedings{jain2013low,
  title={Low-rank matrix completion using alternating minimization},
  author={Jain, P. and Netrapalli, P. and Sanghavi, S.},
  booktitle={ACM symposium on Theory of computing},
  pages={665--674},
  year={2013},
}

@article{SequentialCS08,
title={Sequential Compressed Sensing},
author={Malioutov, D.M. and Sanghavi, S. and Willsky, A.S.},
journal={to appear in IEEE Journal on Special Topics in Signal Processing},
year={2010},
month = {December},
volume={},
number={},
pages={3357-3360},
}

@ARTICLE{CanTaoDecode05,
title={Decoding by linear programming},
author={Cand\`es, E. J. and Tao, T.},
journal={IEEE Transactions on Information Theory},
year={2005},
month={Dec.},
volume={51},
number={12},
pages={ 4203-4215},
}

@inproceedings{cherapanamjeri2017thresholding,
  title={Thresholding Based Outlier Robust {PCA}},
  author={Cherapanamjeri, Yeshwanth and Jain, Prateek and Netrapalli, Praneeth},
  booktitle={Conference on Learning Theory},
  pages={593--628},
  year={2017}
}

@article{maunu2017well,
  title={A Well-Tempered Landscape for Non-convex Robust Subspace Recovery},
  author={Maunu, Tyler and Zhang, Teng and Lerman, Gilad},
  journal={arXiv preprint arXiv:1706.03896},
  year={2017}
}

@article{jafarpour2009efficient,
  title={Efficient and robust compressed sensing using optimized expander graphs},
  author={Jafarpour, S. and Xu, W. and Hassibi, B. and Calderbank, R.},
  journal={IEEE Transactions on Information Theory},
  volume={55},
  number={9},
  pages={4299--4308},
  year={2009},
  publisher={IEEE}
}

@article{gilbert2010sparse,
  title={Sparse recovery using sparse matrices},
  author={Gilbert, A. and Indyk, P.},
  journal={Proceedings of the IEEE},
  volume={98},
  number={6},
  pages={937--947},
  year={2010},
  publisher={IEEE}
}

@article{RecFazPar07,
author = {B. Recht and M. Fazel and P. A. Parrilo},
title = {Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization},
publisher = {SIAM},
year = {2010},
journal = {SIAM Review},
volume = {52},
number = {3},
pages = {471-501},
}


@ARTICLE{CanPla10,
author={Cand\`es, Emmanuel and Plan, Yaniv},
journal={Proceedings of the IEEE},
title={Matrix Completion With Noise},
year={2010},
month={June },
volume={98},
number={6},
pages={925 -936},
}

@ARTICLE{CandTao06,
title={Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies?},
author={Cand\`es, E. J. and Tao, T.},
journal={IEEE Transactions on Information Theory},
year={2006},
month={Dec. },
volume={52},
number={12},
pages={5406-5425},
}

@ARTICLE{CandRomTao06,
title={Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information},
author={Cand\`es, E. J. and Romberg, J. and Tao, T.},
journal={IEEE Transactions on Information Theory},
year={2006},
month={Feb.},
volume={52},
number={2},
pages={ 489-509},
}


@article{chandrasekaran2011rank,
  title={Rank-sparsity incoherence for matrix decomposition},
  author={Chandrasekaran, Venkat and Sanghavi, Sujay and Parrilo, Pablo A and Willsky, Alan S},
  journal={SIAM Journal on Optimization},
  volume={21},
  number={2},
  pages={572--596},
  year={2011},
  publisher={SIAM}
}

@article{ahmed2014blind,
  title={Blind deconvolution using convex programming},
  author={Ahmed, Ali and Recht, Benjamin and Romberg, Justin},
  journal={IEEE Transactions on Information Theory},
  volume={60},
  number={3},
  pages={1711--1732},
  year={2014},
  publisher={IEEE}
}

@article{CanLiMaWri09,
 author = {Cand\`{e}s, Emmanuel and Li, Xiaodong and Ma, Yi and Wright, John},
 title = {Robust principal component analysis?},
 journal = {Journal of ACM},
 issue_date = {May 2011},
 volume = {58},
 number = {3},
 month = {Jun},
 year = {2011},
 pages = {11:1--11:37},
}

@article{prony1795essai,
  title={Essai experimental et analytique},
  author={Prony, R.},
  journal={J. de l'Ecole Polytechnique (Paris)},
  volume={1},
  number={2},
  pages={24--76},
  year={1795}
}



@article{narayan2012atomic,
  title={Atomic norm denoising with applications to line spectral estimation},
  author={Narayan Bhaskar, B. and Tang, G. and Recht, B.},
  url = {http://arxiv.org/abs/1204.0562},
  year={2012}
}


@article{Negahban2012restricted,
  title={Restricted strong convexity and weighted matrix completion: Optimal bounds with noise},
  author={Negahban, S. and Wainwright, M.J.},
  journal={Journal of Machine Learning Research},
  pages={1665--1697},
  month = {May},
  year={2012},
}

@article{grant2008cvx,
  title={{CVX}: Matlab software for disciplined convex programming},
  author={Grant, M. and Boyd, S. and Ye, Y.},
  journal={Online accessiable: http://stanford. edu/\~{} boyd/cvx},
  year={2008}
}


@ARTICLE{MekJaiDhi2009,
title={Guaranteed Rank Minimization via Singular Value Projection},
author={Raghu Meka and Prateek Jain and Inderjit S. Dhillon},
journal={preprint},
year={2009},
url = {http://arxiv.org/abs/0909.5457}
}

@article{CanPla2009,
author={Cand\`es, E.J. and Plan, Y.},
journal={47th Annual Allerton Conference on Communication, Control, and Computing}, title={Accurate low-rank matrix recovery from a small number of linear measurements},
year={2009},
month={September},
volume={},
number={},
pages={1223 -1230},
}

@ARTICLE{YinBaiKri88,
title={On the limit of the largest eigenvalue of the large dimensional sample covariance matrix},
author={Y. Q. Yin and Z. D. Bai and P. R. Krishnaiah},
journal={Probability Theory and Related Fields},
year={1988}}

@ARTICLE{CanRomTao06,
author={Cand\`es, E.J. and Romberg, J. and Tao, T.},
journal={IEEE Transactions on Information Theory},
title={Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information},
year={2006},
month={February},
volume={52},
number={2},
pages={ 489 - 509},
keywords={ Fourier coefficient; convex optimization; discrete-time signal; image reconstruction; incomplete frequency information; linear programming; minimization problem; nonlinear sampling theorem; piecewise constant object; probability value; robust uncertainty principle; signal reconstruction; sparse random matrix; trigonometric expansion; Fourier analysis; convex programming; image reconstruction; image sampling; indeterminancy; linear programming; minimisation; piecewise constant techniques; probability; signal reconstruction; signal sampling; sparse matrices;},
}

@ARTICLE{CanTao2006,
author={Cand\`es, E.J. and Tao, T.},
journal={IEEE Transactions on Information Theory}, title={Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies?},
year={2006},
month={December },
volume={52},
number={12},
pages={5406 -5425},
}

@ARTICLE{Don2006,
author={Donoho, D.L.},
journal={IEEE Transactions on Information Theory},
title={Compressed sensing},
year={2006},
month={April },
volume={52},
number={4},
pages={1289 -1306},
}

@article{lu2017phase,
  title={Phase transitions of spectral initialization for high-dimensional nonconvex estimation},
  author={Lu, Yue M and Li, Gen},
  journal={arXiv preprint arXiv:1702.06435},
  year={2017}
}

@article{shechtman2013gespar,
  title={{GESPAR}: Efficient phase retrieval of sparse signals},
  author={Shechtman, Y. and Beck, A. and Eldar, Y. C.},
  journal={IEEE Transactions on Signal Processing},
  volume={62},
  number={4},
  pages={928--938},
  year={2014},
  publisher={IEEE}
}

@ARTICLE{CanTao10,
author={Cand\`es, Emmanuel and Tao, Terence},
journal={IEEE Transactions on Information Theory},
title={The Power of Convex Relaxation: Near-Optimal Matrix Completion},
year={2010},
month={May },
volume={56},
number={5},
pages={2053 -2080},
}

@ARTICLE{KesMonSew2010,
author={Keshavan, R. H. and Montanari, A. and Oh, S.},
journal={IEEE Transactions on Information Theory}, title={Matrix Completion From a Few Entries},
year={2010},
month={June },
volume={56},
number={6},
pages={2980 -2998},
}

@article{cambareri2016non,
  title={A Non-Convex Blind Calibration Method for Randomised Sensing Strategies},
  author={Cambareri, Valerio and Jacques, Laurent},
  journal={arXiv preprint arXiv:1605.02615},
  year={2016}
}

@inproceedings{li2017blind,
  title={Blind gain and phase calibration for low-dimensional or sparse signal sensing via power iteration},
  author={Li, Yanjun and Lee, Kiryung and Bresler, Yoram},
  booktitle={Sampling Theory and Applications (SampTA), 2017 International Conference on},
  pages={119--123},
  year={2017},
  organization={IEEE}
}

@article{LeeBre2009,
author={Bresler, Y. and Kiryung Lee},
journal={IEEE International Symposium on Information Theory}, title={Efficient and guaranteed rank minimization by atomic decomposition},
year={2009},
month={June},
volume={},
number={},
pages={314 -318},
}

@ARTICLE{GolMa2009,
author={Shiqian Ma and Donald Goldfarb  and Lifeng Chen},
title={Fixed point and Bregman iterative methods for matrix rank minimization},
month = {September},
year = {2009},
journal = {Mathematical Programming}
}

@ARTICLE{TroGil07,
author={Tropp, J.A. and Gilbert, A.C.},
journal={IEEE Transactions on Information Theory}, title={Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit},
year={2007},
month={Dec. },
volume={53},
number={12},
pages={4655 -4666},
}



@ARTICLE{DonohoHuo01,
author={Donoho, D.L. and Huo, X.},
journal={IEEE Transactions on Information Theory}, title={Uncertainty principles and ideal atomic decomposition},
year={2001},
month={November},
volume={47},
number={7},
pages={2845 -2862},
}

@ARTICLE{EladBruckstein02,
author={Elad, M. and Bruckstein, A.M.},
journal={IEEE Transactions on Information Theory},
title={A generalized uncertainty principle and sparse representation in pairs of bases},
year={2002},
month={Sep},
volume={48},
number={9},
pages={ 2558 - 2567},
}

@ARTICLE{Schniter2015,
author={Schniter, P. and Rangan, S.},
journal={IEEE Transactions on Signal Processing},
title={Compressive Phase Retrieval via Generalized Approximate Message Passing},
year={2015},
month={Feb},
volume={63},
number={4},
pages={1043-1055},
keywords={Gaussian processes;compressed sensing;image retrieval;intensity measurement;message passing;CPRL;GESPAR algorithms;PR-GAMP algorithm;SNR;blurred Fourier intensity measurement;compressive phase retrieval;generalized approximate message passing;grayscale image recovery;linear measurement;phase transition;probabilistic approach;sparse Bernoulli-Gaussian signal recovery;Approximation algorithms;Belief propagation;Damping;Noise;Noise measurement;Phase measurement;Signal processing algorithms;Belief propagation;compressed sensing;estimation;phase retrieval},
doi={10.1109/TSP.2014.2386294},
ISSN={1053-587X},}

@ARTICLE{NegRavWaiYu09,
author={Sahand Negahban and Pradeep Ravikumar and Martin Wainwright and Bin Yu},
journal={NIPS 2009},
title={A unified framework for high-dimensional analysis of {M}-estimators with Decomposable Regularizers},
month={December},
year = {2009}
}

@book{Datt08,
author = {Jon Dattorro},
title={Convex Optimization \& {E}uclidean Distance Geometry},
year = {2008},
publisher = {Meboo Publishing}
}

@article{XuHas08,
author={Weiyu Xu and Hassibi, B.},
booktitle={Communication, Control, and Computing, 2008 46th Annual Allerton Conference on}, title={Compressed sensing over the Grassmann manifold: A unified analytical framework},
year={2008},
month={23-26},
volume={},
number={},
pages={562 -567},
}

@article{ArgEvgPon08,
 author = {Argyriou, Andreas and Evgeniou, Theodoros and Pontil, Massimiliano},
 title = {Convex multi-task feature learning},
 journal = {Machine Learning},
 volume = {73},
 number = {3},
 year = {2008},
 pages = {243--272},
 publisher = {Kluwer Academic Publishers},
 address = {Hingham, MA, USA},
 }

@ARTICLE{Wai2009,
author={Wainwright, M.J.},
journal={IEEE Transactions on Information Theory}, title={Sharp Thresholds for High-Dimensional and Noisy Sparsity Recovery Using Constrained Quadratic Programming ({Lasso})},
year={2009},
month={May },
volume={55},
number={5},
pages={2183 -2202},
}

@article{GargKhan09,
 author = {Garg, Rahul and Khandekar, Rohit},
 title = {Gradient descent with sparsification: an iterative algorithm for sparse recovery with restricted isometry property},
 journal = {Proceedings of the 26th Annual International Conference on Machine Learning},
 series = {ICML '09},
 year = {2009},
 isbn = {978-1-60558-516-1},
 location = {Montreal, Quebec, Canada},
 pages = {337--344},
 numpages = {8},
 publisher = {ACM},
 address = {New York, NY, USA},
}

@article{BluDav09,
title = "Iterative hard thresholding for compressed sensing",
journal = "Applied and Computational Harmonic Analysis",
volume = "27",
number = "3",
pages = "265 - 274",
year = "2009",
note = "",
author = "Thomas Blumensath and Mike E. Davies",
}

@ARTICLE{BluDav09IT,
author={Blumensath, T. and Davies, M.E.},
journal={IEEE Transactions on Information Theory},
title={Sampling Theorems for Signals From the Union of Finite-Dimensional Linear Subspaces},
year={2009},
month={April },
volume={55},
number={4},
pages={1872 -1882},
}

@article{bandeira2014saving,
  title={Saving phase: Injectivity and stability for phase retrieval},
  author={Bandeira, A. S. and Cahill, J. and Mixon, D. G. and Nelson, A. A.},
  journal={Applied and Computational Harmonic Analysis},
  volume={37},
  number={1},
  pages={106--125},
  year={2014},
}

@article{eldar2014phase,
  title={Phase retrieval: Stability and recovery guarantees},
  author={Eldar, Y. C. and Mendelson, S.},
  journal={Applied and Computational Harmonic Analysis},
  volume={36},
  number={3},
  pages={473--494},
  year={2014},
}

@ARTICLE{EldMis09,
author={Eldar, Y.C. and Mishali, M.},
journal={IEEE Transactions on Information Theory}, title={Robust Recovery of Signals From a Structured Union of Subspaces},
year={2009},
month={Nov. },
volume={55},
number={11},
pages={5302 -5316},
keywords={block sparse vector recovery;compressed sensing;joint sparsity pattern;multiple measurement vector recovery;nonlinear signal model;restricted isometry property;sampling functions;sampling theory;signal reconstruction;signal recovery;subspace structured union;signal reconstruction;signal sampling;},
doi={10.1109/TIT.2009.2030471},
ISSN={0018-9448},}

@ARTICLE{OboWaiJor10,
author={G. Obozinski and M. J. Wainwright and M. I. Jordan},
journal={to appear in Annals of Statistics},
title={Support union recovery in high-dimensional multivariate regression},
year={2010},
}

@ARTICLE{NegWai09,
author={S. Negahban and M. J. Wainwright.},
journal={UC Berkeley Technical Report 774},
title={Simultaneous support recovery in high dimensions: Benefits and perils of block $\ell_1/\ell_\infty$-regularization.},
year={2009},
month = {May},
url = {http://arxiv.org/abs/0905.0642}
}

@article{Tib96,
     title = {Regression Shrinkage and Selection via the Lasso},
     author = {Tibshirani, Robert},
     journal = {Journal of the Royal Statistical Society. Series B (Methodological)},
     volume = {58},
     number = {1},
     jstor_formatteddate = {1996},
     pages = {267--288},
     year = {1996},
     publisher = {Blackwell Publishing for the Royal Statistical Society},
    }

@article{CheDonSau01,
     title = {Atomic Decomposition by Basis Pursuit},
     author = {Chen, Shaobing and Donoho, David L. and Saunders, Michael A.},
     journal = {SIAM Review},
     volume = {43},
     number = {1},
     pages = {129-159},
     year = {2001},
     publisher = {Society for Industrial and Applied Mathematics},
     }

@article{ZhaoYu06,
     title = {On Model Selection Consistency of Lasso},
     author = {P. Zhao and Bin Yu},
     journal = {Journal of Machine Learning Research},
     volume = {7},
     pages = {2541-2563},
     month = {Nov.},
     year = {2006},
     }

@article{XuCarMan10,
     title = {Principal Component Analysis with Contaminated Data: The High Dimensional Case},
     author = {H. Xu and C. Caramanis and S. Mannor},
     journal = {The 23rd Annual Conference on Learning Theory (COLT)},
     month = {June},
     year = {2010},
     }

@inproceedings{bennett2007netflix,
  title={The netflix prize},
  author={Bennett, James and Lanning, Stan},
  year={2007}
}








@article{efron2010correlated,
  title={Correlated z-values and the accuracy of large-scale statistical estimates},
  author={Efron, Bradley},
  journal={Journal of the American Statistical Association},
  volume={105},
  number={491},
  pages={1042--1055},
  year={2010},
  publisher={Taylor \& Francis}
}


@article{efron2007correlation,
  title={Correlation and large-scale simultaneous significance testing},
  author={Efron, Bradley},
  journal={Journal of the American Statistical Association},
  volume={102},
  number={477},
  pages={93--103},
  year={2007},
  publisher={Taylor \& Francis}
}

@article{fan2012estimating,
  title={Estimating false discovery proportion under arbitrary covariance dependence},
  author={Fan, Jianqing and Han, Xu and Gu, Weijie},
  journal={Journal of the American Statistical Association},
  volume={107},
  number={499},
  pages={1019--1035},
  year={2012},
  publisher={Taylor \& Francis Group}
}

@article{fan2019farmtest,
  title={FarmTest: Factor-Adjusted Robust Multiple Testing with Approximate False Discovery Control},
  author={Fan, Jianqing and Ke, Yuan and Sun, Qiang and Zhou, Wen-Xin},
  journal={Journal of American Statistical Association},
  year={2019+}
}

@article{kneip2011factor,
  title={Factor models and variable selection in high-dimensional regression analysis},
  author={Kneip, Alois and Sarda, Pascal},
  journal={The Annals of Statistics},
  volume={39},
  number={5},
  pages={2410--2447},
  year={2011},
  publisher={Institute of Mathematical Statistics}
}

@article{paul2008preconditioning,
  title={"Preconditioning" for feature selection and regression in high-dimensional problems},
  author={Paul, Debashis and Bair, Eric and Hastie, Trevor and Tibshirani, Robert},
  journal={The Annals of Statistics},
  volume={36},
  number={4},
  pages={1595--1618},
  year={2008},
  publisher={Institute of Mathematical Statistics}
}

@article{fan2018factor,
  title={Factor-Adjusted Regularized Model Selection},
  author={Fan, Jianqing and Ke, Yuan and Wang, Kaizheng},
  journal={arXiv preprint arXiv:1612.08490},
  year={2018}
}

@article{bai2006confidence,
  title={Confidence intervals for diffusion index forecasts and inference for factor-augmented regressions},
  author={Bai, Jushan and Ng, Serena},
  journal={Econometrica},
  volume={74},
  number={4},
  pages={1133--1150},
  year={2006},
  publisher={Wiley Online Library}
}

@article{jolliffe1982note,
  title={A note on the use of principal components in regression},
  author={Jolliffe, Ian T},
  journal={Journal of the Royal Statistical Society: Series C (Applied Statistics)},
  volume={31},
  number={3},
  pages={300--303},
  year={1982},
  publisher={Wiley Online Library}
}

@article{fan2017sufficient,
  title={Sufficient forecasting using factor models},
  author={Fan, Jianqing and Xue, Lingzhou and Yao, Jiawei},
  journal={Journal of econometrics},
  volume={201},
  number={2},
  pages={292--306},
  year={2017},
  publisher={Elsevier}
}

@article{fan2013large,
  title={Large covariance estimation by thresholding principal orthogonal complements},
  author={Fan, J. and Liao, Y. and Mincheva, M.},
  journal={Journal of the Royal Stat.~Society:~Series B},
  volume={75},
  number={4},
  pages={603--680},
  year={2013},
}


@article{fan2019robust,
  title={Robust covariance estimation for approximate factor models},
  author={Fan, Jianqing and Wang, Weichen and Zhong, Yiqiao},
  journal={Journal of econometrics},
  volume={208},
  number={1},
  pages={5--22},
  year={2019},
  publisher={Elsevier}
}

@book{shao2003mathematical,
	title={Mathematical Statistics},
	author={Shao, J.},
	isbn={9780387953823},
	lccn={98045794},
	series={Springer Texts in Statistics},
	url={https://books.google.com/books?id=1HQmBKMgDaMC},
	year={2003},
	publisher={Springer}
}

@article{battey2018distributed,
  title={Distributed testing and estimation under sparse high dimensional models},
  author={Battey, Heather and Fan, Jianqing and Liu, Han and Lu, Junwei and Zhu, Ziwei},
  journal={Annals of statistics},
  volume={46},
  number={3},
  pages={1352},
  year={2018},
  publisher={NIH Public Access}
}

@inproceedings{ganesh2010dense,
  title={Dense error correction for low-rank matrices via principal component pursuit},
  author={Ganesh, Arvind and Wright, John and Li, Xiaodong and Candes, Emmanuel J and Ma, Yi},
  booktitle={2010 IEEE international symposium on information theory},
  pages={1513--1517},
  year={2010},
  organization={IEEE}
}

@article{agarwal2012noisy,
  title={Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions},
  author={Agarwal, Alekh and Negahban, Sahand and Wainwright, Martin J},
  journal={The Annals of Statistics},
  volume={40},
  number={2},
  pages={1171--1197},
  year={2012},
  publisher={Institute of Mathematical Statistics}
}

@article{aybat2014efficient,
  title={Efficient algorithms for robust and stable principal component pursuit problems},
  author={Aybat, Necdet Serhat and Goldfarb, Donald and Ma, Shiqian},
  journal={Computational Optimization and Applications},
  volume={58},
  number={1},
  pages={1--29},
  year={2014},
  publisher={Springer}
}

@inproceedings{xu2010robust,
  title={Robust PCA via outlier pursuit},
  author={Xu, Huan and Caramanis, Constantine and Sanghavi, Sujay},
  booktitle={Advances in Neural Information Processing Systems},
  pages={2496--2504},
  year={2010}
}

@article{ma2018efficient,
  title={Efficient optimization algorithms for robust principal component analysis and its variants},
  author={Ma, Shiqian and Aybat, Necdet Serhat},
  journal={Proceedings of the IEEE},
  volume={106},
  number={8},
  pages={1411--1426},
  year={2018},
  publisher={IEEE}
}

@article{vaswani2018static,
  title={Static and dynamic robust PCA and matrix completion: A review},
  author={Vaswani, Namrata and Narayanamurthy, Praneeth},
  journal={Proceedings of the IEEE},
  volume={106},
  number={8},
  pages={1359--1379},
  year={2018},
  publisher={IEEE}
}

@article{qiu2014recursive,
  title={Recursive robust pca or recursive sparse recovery in large but structured noise},
  author={Qiu, Chenlu and Vaswani, Namrata and Lois, Brian and Hogben, Leslie},
  journal={IEEE Transactions on Information Theory},
  volume={60},
  number={8},
  pages={5007--5039},
  year={2014},
  publisher={IEEE}
}

@inproceedings{qiu2010real,
  title={Real-time robust principal components' pursuit},
  author={Qiu, Chenlu and Vaswani, Namrata},
  booktitle={2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton)},
  pages={591--598},
  year={2010},
  organization={IEEE}
}

@article{guo2014online,
  title={An online algorithm for separating sparse and low-dimensional signal sequences from their sum},
  author={Guo, Han and Qiu, Chenlu and Vaswani, Namrata},
  journal={IEEE Transactions on Signal Processing},
  volume={62},
  number={16},
  pages={4284--4297},
  year={2014},
  publisher={IEEE}
}

@book{bremaud2013markov,
  title={Markov chains: Gibbs fields, Monte Carlo simulation, and queues},
  author={Br{\'e}maud, Pierre},
  volume={31},
  year={2013},
  publisher={Springer Science \& Business Media}
}

@article{hsu2011robust,
  title={Robust matrix decomposition with sparse corruptions},
  author={Hsu, Daniel and Kakade, Sham M and Zhang, Tong},
  journal={IEEE Transactions on Information Theory},
  volume={57},
  number={11},
  pages={7221--7234},
  year={2011},
  publisher={IEEE}
}

@article{JMLR:v18:16-140,
  author  = {Jianqing Fan and Weichen Wang and Yiqiao Zhong},
  title   = {An $\ell_{\infty}$ Eigenvector Perturbation Bound and Its Application},
  journal = {Journal of Machine Learning Research},
  year    = {2018},
  volume  = {18},
  number  = {207},
  pages   = {1-42},
  url     = {http://jmlr.org/papers/v18/16-140.html}
}

@article{fan2018robust,
  title={Robust high dimensional factor models with applications to statistical machine learning},
  author={Fan, Jianqing and Wang, Kaizheng and Zhong, Yiqiao and Zhu, Ziwei},
  journal={arXiv preprint arXiv:1808.03889},
  year={2018}
}

@article{fan2008high,
  title={High dimensional covariance matrix estimation using a factor model},
  author={Fan, Jianqing and Fan, Yingying and Lv, Jinchi},
  journal={Journal of Econometrics},
  volume={147},
  number={1},
  pages={186--197},
  year={2008},
  publisher={Elsevier}
}

@article{wong2017matrix,
  title={Matrix completion with noisy entries and outliers},
  author={Wong, Raymond KW and Lee, Thomas},
  journal={The Journal of Machine Learning Research},
  volume={18},
  number={1},
  pages={5404--5428},
  year={2017},
  publisher={JMLR. org}
}

@article {Chen22931,
	author = {Chen, Yuxin and Fan, Jianqing and Ma, Cong and Yan, Yuling},
	title = {Inference and uncertainty quantification for noisy matrix completion},
	volume = {116},
	number = {46},
	pages = {22931--22937},
	year = {2019},
	doi = {10.1073/pnas.1910053116},
	publisher = {National Academy of Sciences},
	issn = {0027-8424},
	URL = {https://www.pnas.org/content/116/46/22931},
	eprint = {https://www.pnas.org/content/116/46/22931.full.pdf},
	journal = {Proceedings of the National Academy of Sciences}
}

@article{klopp2017robust,
  title={Robust matrix completion},
  author={Klopp, Olga and Lounici, Karim and Tsybakov, Alexandre B},
  journal={Probability Theory and Related Fields},
  volume={169},
  number={1-2},
  pages={523--564},
  year={2017},
  publisher={Springer}
}

@inproceedings{cherapanamjeri2017nearly,
  title={Nearly optimal robust matrix completion},
  author={Cherapanamjeri, Yeshwanth and Gupta, Kartik and Jain, Prateek},
  booktitle={Proceedings of the 34th International Conference on Machine Learning-Volume 70},
  pages={797--805},
  year={2017},
  organization={JMLR. org}
}

@article{zhang2014novel,
  title={A novel m-estimator for robust pca},
  author={Zhang, Teng and Lerman, Gilad},
  journal={The Journal of Machine Learning Research},
  volume={15},
  number={1},
  pages={749--808},
  year={2014},
  publisher={JMLR. org}
}

@inproceedings{ha2015robust,
  title={Robust pca with compressed data},
  author={Ha, Wooseok and Barber, Rina Foygel},
  booktitle={Advances in Neural Information Processing Systems},
  pages={1936--1944},
  year={2015}
}

@article{tomasi1992shape,
  title={Shape and motion from image streams under orthography: a factorization method},
  author={Tomasi, Carlo and Kanade, Takeo},
  journal={International journal of computer vision},
  volume={9},
  number={2},
  pages={137--154},
  year={1992},
  publisher={Springer}
}
