Online NMF with Outliers
Renbo~Zhao
~\IEEEmembership{Member
~IEEE
We propose a unified and systematic framework for performing online nonnegative matrix factorization in the presence of outliers. Our framework is particularly suited to large-scale data. We propose two solvers based on projected gradient descent and the alternating direction method of multipliers. We prove that the sequence of objective values converges almost surely by appealing to the quasi-martingale convergence theorem. We also show the sequence of learned dictionaries converges to the set of stationary points of the expected loss function almost surely. In addition, we extend our basic problem formulation to various settings with different constraints and regularizers. We also adapt the solvers and analyses to each setting. We perform extensive experiments on both synthetic and real datasets. These experiments demonstrate the computational efficiency and efficacy of our algorithms on tasks such as (parts-based) basis learning, image denoising, shadow removal and foreground-background separation. %in terms of the peak signal-to-noise ratio, the interpretability of the basis matrix, image denoising and shadow removal capabilities, and finally, the computational speed.
merged.tex
%\documentclass[10pt,onecolumn,journal]{IEEEtran}
\documentclass[10pt,twocolumn,journal]{IEEEtran}
%\documentclass[10pt]{article}
%\documentclass[journal]{IEEEtran}

%\usepackage{fullpage}
%\usepackage[letterpaper,top=1in,bottom=1in,left=1in,right=1in]{geometry}
%\usepackage{xr}
\usepackage{cite}
\usepackage[colorlinks=true,linkcolor=black,citecolor=black,urlcolor=black]{hyperref}
%\externaldocument{to_supp}
%\DeclareCaptionFormat{lower}{#1#2\lowercase{#3}\par}
%\usepackage{setspace}
%\doublespacing
%\singlespacing

\input{preamble}
\graphicspath{{../figures/}}
\newcommand{\rmth}{\mathrm{th}}

\begin{document}

\title{Online Nonnegative Matrix Factorization with Outliers}
%\title{Online NMF with Outliers}

\author{Renbo~Zhao,~\IEEEmembership{Member,~IEEE,}
        and~Vincent~Y.~F.~Tan,~\IEEEmembership{Senior~Member,~IEEE}
        
\thanks{A preliminary work has been published in ICASSP 2016 \cite{Zhao_16}.} 

\thanks{The authors  are with the Department of Electrical and Computer Engineering and the Department of Mathematics, National University of Singapore. They are supported in part by the NUS Young Investigator Award (grant number R-263-000-B37-133).}
}
%\thanks{M. Shell was with the Department
%of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta,
%GA, 30332 USA e-mail: (see http://www.michaelshell.org/contact.html).}% <-this % stops a space
%\thanks{J. Doe and J. Doe are with Anonymous University.}% <-this % stops a space
%\thanks{Manuscript received April 19, 2005; revised August 26, 2015.}}

%\markboth{IEEE Transactions on Signal Processing,~Vol.~, No.~, 2016}%
%{Shell \MakeLowercase{\textit{Zhao et al.}}: Online Nonnegative Matrix Factorization with Outliers}

\maketitle

\begin{abstract}
We propose a unified and systematic framework for performing online nonnegative matrix factorization in the presence of outliers. Our framework is particularly suited to large-scale data.  We propose two solvers  based on   projected gradient descent and  the  alternating direction method of multipliers. We prove that the sequence of objective values converges almost surely by appealing to the quasi-martingale convergence theorem. We also show the sequence of learned dictionaries converges to the set of stationary points of the expected loss function almost surely.  
In addition, we extend our basic problem formulation to various settings with different constraints and regularizers. We also adapt the solvers and analyses to each setting. 
We perform extensive experiments on both synthetic and real datasets. These experiments demonstrate the computational efficiency and efficacy of our algorithms  on tasks such as (parts-based) basis learning, image denoising, shadow removal and foreground-background separation. 

%in terms of the peak signal-to-noise ratio, the interpretability of the basis matrix, image denoising and shadow removal capabilities, and finally, the computational speed.
\end{abstract}

 
\begin{IEEEkeywords}
 Nonnegative matrix factorization, Online learning, Robust learning,  Projected gradient descent, Alternating direction method of multipliers 
 \end{IEEEkeywords}



\section{Introduction}\label{sec:intro}
In recent years, Nonnegative Matrix Factorization (NMF) has become a popular dimensionality reduction \cite{Tsuge_01} technique, due to its parts-based, non-subtractive  interpretation of the learned basis \cite{Lee_99}. Given a nonnegative data matrix $\bV$, it seeks to approximately decompose $\bV$ into two nonnegative matrices, $\bW$ and $\bH$, such that $\bV\approx\bW\bH$. In the literature, the fidelity of such a approximation is most commonly measured by $\norm{\bV-\bW\bH}_F^2$ \cite{Lee_00,Kim_08a,Lin_07a,Kim_08b,Xu_12}. 
To obtain this approximation, many algorithms have been proposed, including multiplicative updates \cite{Lee_00}, block principal pivoting \cite{Kim_08a}, projected gradient descent \cite{Lin_07a}, active set method\cite{Kim_08b}, and the alternating direction method of multipliers \cite{Xu_12}. These algorithms have promising performances in numerous applications, including document clustering \cite{Ding_05}, hyperspectral unmixing \cite{Yuan_15} and audio source separation \cite{Durrieu_11}. However, there are also many studies \cite{Cao_07,Zhang_11} showing that their performances deteriorate under two common and practical scenarios. The first scenario is when the data matrix $\bV$ has a large number of columns (data samples). This situation arises in today's data-rich environment. {\em Batch} data processing methods used in  the aforementioned algorithms become  highly inefficient  in terms of  the computational time and storage space.  
The second scenario is the existence of outliers in some of the data samples. For example (e.g.), there are  glares and shadows in images   due to bad illumination conditions. Another example is the presence of  impulse noises in time series,  including speech recordings in natural language  processing or temperature recordings in weather forecasting. The outliers, if not handled properly, can significantly corrupt the learned basis matrix, thus the underlying low-dimensional data structure cannot be learned reliably. Moreover, outlier detection and pruning become much more difficult in large-scale datasets~\cite{RAD_15}. As such, it is imperative to design   algorithms that can learn interpretable parts-based basis representations from large-scale datasets whilst being robust to possible outliers.

\subsection{Previous Works} \label{sec:prev_work}
Many efforts have been devoted to address each challenge separately. To handle large datasets,  researchers have pursued solutions in three main directions. The first class of algorithms proposed is know as {\em online NMF} algorithms \cite{Cao_07,Bucak_08,Guan_12,Lef_11,WangFei_11,WuShen_14}. These algorithms aim to refine the basis matrix each time a new data sample is acquired without storing the past data samples. The second class of algorithms is known as {\em distributed NMF}\cite{Liu_10,Gemulla_11,Du_14,Chen_15}. The basic idea behind these algorithms is to distribute the data samples over a network of agents so that several small-scale optimization problems can be performed concurrently. 
The final class of algorithms is called the {\em compressed NMF} algorithms \cite{Halko_11,Tepper_16}. These algorithms perform structured random compression to %to reduce the size of the dataset. In \cite{Tepper_16}, the authors performed  random projections,  %exploit the data structure and 
project the data onto the lower-dimensional manifolds. As such, the size of the dataset can be reduced. 
These three approaches have successfully reduced the computation and storage complexities---either provably or through numerical experiments.  
For the existence of outliers, a class of algorithms called the (batch) {\em robust NMF} \cite{Gao_15, Huang_14, Yang_13, Du_12, Ding_12, Ding_11, Cichoc_11, Kasi_12,Fev_14,Shen_14,Zhang_11} has been proposed to reliably learn the basis matrix by minimizing the effects of the outliers. The robustness against the outliers is achieved via different approaches. These are detailed in Section~\ref{sec:RNMF}. However, to the best of our knowledge, so far there are no NMF-based algorithms that are able to systematically handle  outliers in large-scale datasets.

\subsection{Main Contributions}
In this paper, we propose an algorithm called the {\em online NMF with outliers} that fills this void. Specifically, our algorithm aims to learn the basis matrix $\bW$ in an online manner whilst being robust to outliers.   %This makes the algorithm particularly suitable on the large datasets corrupted by the outliers.
The development of the proposed algorithm involves much more than the straightforward combination or adaptation of online NMF and robust NMF algorithms. Indeed, since there are many ways to ``robustify'' the NMF algorithms, it is crucial to find an appropriate way to incorporate such robustness guarantees into the online algorithms. Our algorithm proceeds as follows. At each time instant, we solve two optimization problems. The first enables us to learn the coefficient and outlier vectors while the second enables us to update the basis matrix. %Compared canonical online NMF algorithms, these two optimization problems are more involved. 
We propose two solvers  based on projected gradient descent (PGD) and alternating direction method of multipliers (ADMM) to solve both optimization problems. 
 Moreover, the presence of outliers also results in more difficulty when we analyze the convergence properties of  our  algorithms. See Section~\ref{sec:conv_dis} for a detailed discussion. We remark that in recent years, some algorithms of similar flavors have been proposed, e.g., online robust PCA \cite{Feng_13,Shen_14b} and online robust dictionary learning \cite{Wang_13}. However, due to different problem formulations, our algorithm  has many distinctive features, which then calls for different techniques to develop the solvers and analyze the  convergence properties. Furthermore, in Section~\ref{sec:numericals}, we also observe its superior performance on real-world applications, including (parts-based) basis learning, image denoising, shadow removal and foreground-background separation, over the similar algorithms.  In sum, our contributions are threefold:
\begin{enumerate}
\item We develop two different  solvers  based on PGD and ADMM  to solve the optimization problems  at each time instant.  These two solvers can be easily extended to two novel solvers for the batch robust NMF problem. The theoretical and empirical performances of both solvers are compared and contrasted. 
\item Assuming the data are independently drawn from some common distribution $\bbP$, we prove the almost sure convergence of the sequence of objective values  as well as the almost sure convergence of %each subsequential limit of the sequence of the basis matrices to a local minimizer of the objective function. 
the sequence of learned basis matrices to the set of stationary points of the expected loss function.
The proof techniques involve the use of tools from convex analysis \cite{Rock_70} and empirical process theory\cite{Vaart_00}, as well as the quasi-martingale convergence theorem \cite[Theorem 9.4 \& Proposition 9.5]{Met_82}.
\item   We extend the basic problem setting to various other general settings, by altering the constraint sets and adding regularizers. We also indicate how to adapt our solvers and analyses to each case. By doing so, the applicability of our algorithms is greatly generalized. 
\end{enumerate}
%in terms of both the theoretical convergence analysis and 

\subsection{Paper Organization}\label{sec:org}
This paper is organized as follows. We first provide a  more detailed literature survey in Section~\ref{sec:priorart}. Next we state a formal formulation of our problem in Section~\ref{sec:prob}. The algorithms are derived in Section~\ref{sec:algo} and their convergence properties are analyzed in Section~\ref{sec:conv_analysis}. In Section~\ref{sec:extension}, we extend our basic problem formulations to a wide variety of settings, and indicate how the solvers and analyses can be adapted to each setting. Finally in Section~\ref{sec:numericals}, we provide extensive experiment results on both synthetic and real data. The results are compared to  those of their batch counterparts and other online matrix factorization algorithms.   We conclude the paper in Section \ref{sec:con} stating some promising avenues for further investigations. 


In this paper, all the lemmas and sections with indices beginning with `S' will appear in the supplemental material. 

%\subsection{Contributions}
%In this paper, we have developed an algorithm that can reliably learn the basis matrix in an online fashion in the presence of the outliers. As shown in Section~\ref{sec:conv_analysis} and \ref{sec:numericals}, our algorithms enjoy both strong theoretical convergence properties and superior practical performances. Our contributions are three-fold. First, we develop two solvers, based on PGD and ADMM respectively, to solve the two optimization problems \eqref{eq:min_hr} and \eqref{eq:min_W} at each time instant. These two solvers can be easily extended to two novel solvers for the batch robust NMF problem. The theoretical and empirical performances of both solvers are compared and contrasted. Second, we prove the almost sure convergence of the objective function under our algorithm, as well as the strong convergence of each subsequential limit of the sequence of the basis matrices. Third, we extend our basic problem setting to various cases, by altering the constraint sets and adding regularizers. We also indicate how to adapt our algorithms and analysis to each case. By doing so, the applicability of our algorithms is greatly generalized. 


\subsection{Notations}
In the following, we use capital boldface letters to denote matrices. For example, the (updated) dictionary/basis matrix at time $t$ is denoted by $\bW_t$.  %In particular, we use $\bV$, $\bW$, $\bH$ and $\bR$ to denote data matrix, dictionary/basis matrix, coefficient matrix and outlier matrix respectively, such that $\bV \approx \bW\bH+\bR$.  $\bV$, $\bW$, $\bH$ and $\bR$ are all nonnegative  with dimensions $F\times N$, $F\times K$, $K\times N$ and $F\times N$ respectively. 
We use $F$ and $K$ to denote the ambient dimension and the (known) latent dimension of data respectively. %Both $F$ and $K$ are assumed to be constants.\footnote{In this work, we do not simultaneously consider the data with high ambient dimensions. An attempt on this problem in the context of dictionary learning with the squared-$\ell_2$ loss has been made in \cite{Mensch_16}.} 
%If the streaming data has finite size, we denote the size by $N$. %and the number of data samples respectively. 
We use lower-case boldface letters to denote vectors. Specifically, at time instant $t$, we denote the acquired sample vector, learned coefficient vector and outlier vector as $\bv_t$, $\bh_t$ and $\br_t$ respectively. For a vector $\bx$, its $i$-th entry is denoted by $x_i$. 
%Specifically, we use $\bv$, $\bw$, $\bh$ and $\br$ to denote the columns of $\bV$, $\bW$, $\bH$ and $\bR$, respectively.  
Given a matrix $\bX$, we denote its $i$-th row as $\bX_{i:}$, $j$-th column as $\bX_{:j}$ and $(i,j)$-th entry by $x_{i,j}$. Moreover, we denote its Frobenius norm by $\norm{\bX}_F$, spectral norm by $\norm{\bX}_2$, $\ell_{1,1}$ norm by $\norm{\bX}_{1,1} \defeq \sum_{i,j} \abs{x_{i,j}}$ %, max norm by $\norm{\bX}_{\max}\defeq\max_{i,j} \abs{x_{i,j}}$ 
and trace by $\tr(\bX)$. 
Inequality $\bx\geq 0$ or $\bX \geq 0$ denotes entry-wise nonnegativity. We use $\lrangle{\cdot}{\cdot}$ to denote the Frobenius inner product between two matrices and $\bone$ the vector with all entries equal to one. For a closed convex nonempty set $\calA$, % in a Euclidean space $\calE$, 
we denote $\calP_\calA$ as the Euclidean projector onto $\calA$. In particular, $\calP_+$ denotes the Euclidean projector onto the nonnegative orthant.  Also, the $\infty$-indicator function of $\calA$, $I_\calA$ is defined as
\begin{equation}
I_\calA(x) \defeq \begin{cases}
0, &x\in\calA\\
\infty, & x\not\in\calA
\end{cases}.
\end{equation}
In particular, $I_+$ denotes the $\infty$-indicator function of the nonnegative orthant. 
For $n\in\bbN$, $[n]:=\{1,2,\ldots,n\}$. Also, $\bbR_+$ denotes the set of nonnegative real numbers. 
%For a matrix $\bA$, we denote its $i$-th row as $\bA_{i:}$ and its $j$-th column as $\bA_{:j}$.
%We use $\tr(\cdot)$ to denote the trace operator. 

%In this paper, we defer all the technical lemmas used in Section~\ref{sec:algo} and \ref{sec:analysis} to the supplemental material. The indices of such lemmas are prepended with `S' to distinguish them from the lemmas in the main text. 
\section{Related Works}\label{sec:priorart}
\subsection{Robust NMF}\label{sec:RNMF}
The canonical NMF problem can be stated as the following minimization problem
\begin{equation}
\min_{\bW\in\calC, \{\bh_i\}_{i=1}^N\ge 0} \quad\frac{1}{N}\sum_{i=1}^N \frac{1}{2}\norm{\bv_i-\bW\bh_i}_2^2, \label{eq:canon_NMF}  
\end{equation}
where $\calC\subseteq\bbR_+^{F\times K}$ denotes the constraint set for $\bW$ and $N$ denotes the number of data samples. In many works~\cite{Paa_94,Lee_00,Hoyer_04}, $\calC$ is set to $\bbR_+^{F\times K}$. For simplicity, we omit   regularizers on $\bW$ and $\{\bh_i\}_{i=1}^N$ at this point. %Compared to the canonical version, robust NMF aims to learn 
Since algorithms for the canonical NMF perform unsatisfactorily when the data contain outliers, robust NMF algorithms have been proposed.  Previous algorithms for   robust NMF fall into two categories. The first category~\cite{Gao_15, Huang_14, Yang_13, Du_12, Ding_12, Ding_11, Cichoc_11, Kasi_12} replaces the (half) squared $\ell_2$ loss $1/2\norm{\bv_i-\bW\bh_i}_2^2$ in \eqref{eq:canon_NMF} with some other robust loss measure $\psi(\cdot\Vert\cdot)$
\begin{equation}
\min_{\bW\in\calC, \{\bh_i\}_{i=1}^N\ge 0} \quad \frac{1}{N}\sum_{i=1}^N  \psi(\bv_i\Vert\bW\bh_i). \label{eq:robust1}
\end{equation}
For example, $\psi(\bv_i\Vert\bW\bh_i)$ can be the $\ell_2$ norm $\norm{\bv_i-\bW\bh_i}_2$ \cite{Ding_11} or the $\ell_1$ norm $\norm{\bv_i-\bW\bh_i}_1$ \cite{Kasi_12}.   %$\norm{\bV-\bW\bH}_{2,1}$ \cite{Ding_11} or $\norm{\bV-\bW\bH}_{1,1}$ \cite{Kasi_12}. 
The second category \cite{Fev_14,Shen_14,Zhang_11} retains the squared $\ell_2$ loss but explicitly models the outlier vectors $\{\br_i\}_{i=1}^N$. Specifically, \eqref{eq:canon_NMF} is reformulated as  % the data generation model is $\bV = \bW\bH+\bR+\bE$, where $\bE$ contains \iid Gaussian noise entries. 
\begin{align}
&\min \quad \frac{1}{N}\sum_{i=1}^N \frac{1}{2}\norm{\bv_i-\bW\bh_i-\br_i}_2^2 + \lambda\, \phi(\bR) \nn\\
&\st \quad \bW\in\calC, \{\bh_i\}_{i=1}^N\ge 0, \bR\in\calQ\label{eq:robust2}
\end{align}
where $\lambda\ge 0$ is the regularization parameter, $\bR = [\br_1,\ldots,\br_N]$ is the outlier matrix, $\phi(\bR)$ is the regularizer on $\bR$ and %on $\{\br_i\}_{i=1}^n$ and 
$\calQ$ is the feasible set of $\bR$. Depending on the assumed sparsity structure of $\bR$, $\phi(\bR)$ can be the $\ell_{2,1}$ norm \cite{Fev_14}, $\ell_{1,2}$ norm\cite{Shen_14} or $\ell_{1,1}$ norm \cite{Zhang_11} of $\bR$. %A unified optimization problem formulation for both types can be stated as
Robust NMF algorithms typically do not admit strong recovery guarantees of the original data matrix %(given by $\bW\bH$) %$\bL\defeq\bW\bH$ 
since neither \eqref{eq:robust1} nor \eqref{eq:robust2} are convex programs. However, as shown empirically, the estimated basis matrix $\hatbW$ represents meaningful parts of the data and the residues of the outliers in the reconstructed matrix $\hatbW\hatbH$ are very small \cite{Shen_14,Fev_14,Zhao_16}. 
%the recovered data matrix $\hatbW\hatbH$ has very little outlier residues . 
%in many real world applications, the estimated basis matrix $\hatbW$ or the data matrix $\hatbW\hatbH$ are empirically good \cite{Shen_14,Fev_14}.
%In contrast, another family of algorithms called {\em robust PCA} \cite{Candes_11} can recover $\bL$ exactly, provided $\bL$ satisfies some technical conditions. %({\em restricted isometry property} for example).
%However, compared to RNMF, it also has several limitations: 1) The technical conditions on $\bL$ are not satisfied for most real data, making exact recovery guarantee essentially void; 2) The convex program leading to the recovery of $\bL$ is inefficient to solve; 3) Parts-based representation can not be directly obtained. Therefore, in this work our algorithm leverages on the family of RNMF algorithms, or more specially, the second type since it has explicit representation of outliers and thus is more amenable to analysis. %, while it has been shown recently that solving 

%\subsection{Robust PCA}
% Compared 


\subsection{Online Matrix Factorization}
Existing algorithms on online matrix factorization belong to two distinct categories.  The first category of algorithms \cite{Guan_12,Mairal_10,Feng_13,Shen_14b,WangFei_11} assumes %the underlying low-dimensional data %structure (subspace or polyhedral cone)
%subspace (or polyhedral cone for the nonnegative coding problems) 
%is static. In most cases, it assumes 
the data samples $\{\bv_t\}_{t\ge 1}$ are generated independently from a time-invariant distribution $\bbP$. Under this assumption, it is possible to provide theoretical guarantees on the convergence of the online stochastic algorithms by leveraging the empirical process theory, % as well as the corresponding limit point, 
as was done  in \cite{Mairal_10,Feng_13,Guan_12,Shen_14b}. These methods have extensive applications, including document clustering \cite{WangFei_11}, image inpainting~\cite{Mairal_10}, face recognition~\cite{Guan_12}, and image annotation \cite{Guan_12}. The second category of algorithms  \cite{WuShen_14,Cao_07,Bucak_08,Lef_11,WangDong_13,Wang_13,Xing_13,Zhang_15,Zhang_15b}, with major applications in visual tracking, assumes %the underlying data  is dynamic. In other words, these works assume the data samples are correlated or the distribution of the data samples, 
that the data generation distribution $\bbP$ is time-varying. Although these assumptions are weaker than those in the first class of algorithms, it is very difficult to provide theoretical guarantees on the convergence of the online algorithms. % in a stochastic setting. %An initial attempt has been made in \cite{Zhan_16,Lois_15}, assuming an auto-regression model for data generation. However, their algorithms and analyses are limited to online robust PCA problems and their assumptions are relatively strong. 
%Indeed, almost all the existing algorithms of the second type lack formal performance guarantees. 
%These methods are mainly applied in visual tracking problems. 
%In this work, we focus on the first type problems and derive an algorithm that enjoy both strong theoretical performance guarantees and superior practical applicabilities. 

\subsection{Online Low-rank and Sparse Modeling}\label{sec:OL_lowrank}
Another related line of works \cite{Zhan_16,Lois_15,Guo_15,Spre_12,Spre_12b} aims to recover the low-rank data matrix $\bL$ and the sparse outlier matrix $\bS$ from their additive mixture $\bM$ in an online fashion. Among these works, \cite{Zhan_16,Lois_15,Guo_15} assume that the sequence of mixture vectors (columns of $\bM$) arrives in a streaming fashion. The authors derive the recovery algorithms based on their proposed models of the sequence of ground-truth data vectors (columns of $\bL$) and outlier vectors (columns of $\bS$). 
The authors of \cite{Spre_12,Spre_12b} adopt a different approach. They allow the number of mixture vectors to be large but finite and use an alternating minimization approach to solve a variant of \eqref{eq:robust2} (with additional Tikhonov regularizers on $\bW$ and $\{\bh_i\}_{i=1}^N$). In learning the coefficient vectors $\{\bh_i\}_{i=1}^N$ (with fixed $\bW$), the authors employ the stochastic gradient descent method, thus ensuring that their algorithms are scalable to large-scale data. 


\section{Problem Formulation}\label{sec:prob}
Following Section~\ref{sec:RNMF}, in this work we explicitly model the outlier vectors as $\{\br_t\}_{t\ge 1}$. %instead of using a robust fitting measure $\psi$)
 %leverages on the second type of RNMF algorithms (described in Section~\ref{sec:RNMF}) 
Also, we assume the data generation distribution $\bbP$ is time-invariant, for ease of the convergence analysis. 
First, for a fixed data sample $\bv$ and a fixed basis matrix $\bW$, define the loss function with respect to (w.r.t.) $\bv$ and $\bW$, $\ell(\bv,\bW)$ as
\begin{align}
\ell(\bv,\bW) \defeq \min_{\bh\ge 0,\br\in\calR} \tilell(\bv,\bW, \bh, \br),\label{eq:ith_loss}
\end{align}
where 
\begin{equation}
\tilell(\bv,\bW, \bh, \br)\defeq\frac{1}{2}\norm{\bv-\bW\bh-\br}_2^2 +\lambda\norm{\br}_1, \label{eq:ith_loss_tilde}
\end{equation}
and $\calR \defeq \{\br\in\bbR^F\,|\,\norm{\br}_\infty\le  M\}$ is the constraint set of the outlier vector $\br$. Here we use the $\ell_1$ regularization to promote entrywise sparsity on $\br$. 

Next, given a finite set of data samples $\{\bv_i\}_{i\in[t]}\iidsim\bbP$,  we define the {\em empirical loss} associated with $\{\bv_i\}_{i\in[t]}$, $f_t(\bW)$ as %\eqref{eq:robust2} amounts to the following optimization problem %minimizing an empirical loss function  over $\bW\in\calC$, where  %solving a constrained empirical loss minimization problem
%\begin{align}
%\min_{\bW\in\calC} \quad \left[f_t(\bW) \defeq  \frac{1}{t}\sum_{i=1}^t\ell(\bv_i,\bW)\right] \label{eq:emp_loss}
%\end{align}
\begin{align}
f_t(\bW) \defeq  \frac{1}{t}\sum_{i=1}^t\ell(\bv_i,\bW) \label{eq:emp_loss}
\end{align}
where $\calC \defeq \left\{\bW\in\bbR_+^{F\times K}\,|\,\norm{\bW_{:i}}_2\le 1, \forall\,i\in[K]\right\}$. % and $f_t(\bW)$ is the {\em empirical loss}. %The loss function w.r.t.\ $\bv_i$, $\ell(\bv_i,\bW)$ is defined as

Following the convention of the online learning literature \cite{Tsypkin_71,Bottou_98,Bottou_08}, instead of minimizing the empirical loss $f_t(\bW)$ in \eqref{eq:emp_loss}, we aim to minimize the {\em expected loss} $f(\bW)$, i.e., % over $\bW\in\calC$, where
%as our objective  
\begin{equation}
\min_{\bW\in\calC} \left[f(\bW) \defeq \bbE_{\bv\sim\bbP}[\ell(\bv,\bW)]\right].  \label{eq:exp_loss}
\end{equation}
In other words, we aim to solve a (non-convex) stochastic program \cite{Shap_07}. Note that by the strong law of large numbers, given any  $\bW\in\calC$, we have
$f_t(\bW) \convas f(\bW)$ as $t\to\infty$.

\begin{remark}
We make three remarks here. First we explain the reasonings behind the choice of the constraint sets $\calC$ and $\calR$.  The set $\calC$ constrains the columns of $\bW$ in the unit (nonnegative) $\ell_2$ ball. This is to prevent the entries of $\bW$ from being unbounded, following the conventions in \cite{Lin_07a,Lin_07b}. The set $\calR$ uniformly bounds the entries of $\br$. This is because in practice, both the underlying data (without outliers) and the observed data are uniformly bounded entrywise. Since we do not require $\br$ to be exactly recovered, this prior information can often improve the estimation of $\br$. For real data, the bound $M>0$ can often be easily chosen. For example, for gray-scale images, $M$ can be chosen as $2^m-1$, where $m$ is the number of bits per pixel. In the case of matrices containing ratings from users with a maximum rating of $\upsilon$, $M$ can be chosen as $\upsilon$. In some scenarios where $M$ is difficult to estimate, we simply set $M=\infty$. As will be shown in Sections~\ref{sec:algo} and \ref{sec:conv_analysis}, the algorithms and analyses developed for finite $M$ can be easily adapted to infinite $M$. 
Second, for the sake of brevity, we omit regularizing $\bW$ and $\bh$. Such regularizations, together with other possible constraint sets of $\bW$ and $\br$ will be discussed in Section~\ref{sec:extension}.
Third, we assume the ambient data dimension $F$, the latent data dimension $K$ and the penalty parameter $\lambda$ in \eqref{eq:ith_loss} are time-invariant parameters.\footnote{In this work, we do not simultaneously consider the data with high ambient dimensions. An attempt on this problem in the context of dictionary learning with the squared-$\ell_2$ loss has been made in \cite{Mensch_16}.}  %i.e., they are not functions of $t$. 
\end{remark}


\section{Algorithms}\label{sec:algo}
To tackle the problem proposed in Section~\ref{sec:prob}, we leverage the  {\em stochastic majorization-minimization (MM)} framework\cite{Mairal_13,Raza_16}, which has been widely used in previous works on online matrix factorization \cite{Mairal_10,Feng_13,Shen_14b,WuShen_14,Cao_07,Lef_11,Bucak_08,WangDong_13,Wang_13}. In essence, such framework decomposes the optimization problem in \eqref{eq:exp_loss} into two steps, namely {\em nonnegative encoding} and {\em dictionary update}. Concretely, at a time instant $t$ $(t\ge 1)$, we first learn the coefficient vector $\bh_t$ and the outlier vector $\br_t$ based on the newly acquired data sample $\bv_t$ and the previous dictionary matrix $\bW_{t-1}$.  Specifically, we solve the following convex optimization problem
\begin{equation}
%(\bh_t,\br_t) = \argmin_{\bh\geq 0,\br\in\calR}\;\; \frac{1}{2}\norm{\bv_t-\bW_{t-1}\bh-\br}_2^2 + \lambda\norm{\br}_1. \label{eq:min_hr}
(\bh_t,\br_t) = \argmin_{\bh\geq 0,\br\in\calR}\;\; \tilell(\bv_t,\bW_{t-1}, \bh, \br).\label{eq:min_hr}
\end{equation}
Here the initial basis matrix $\bW_0$ is randomly chosen in $\calC$. 
%where
%\begin{equation}
%\calR \defeq \{\br\in\bbR^F\,|\,\norm{\br}_\infty\le  M\}.
%\end{equation}
Next, based on the past statistics $\{\bv_i,\bh_i,\br_i\}_{i\in[t]}$, the basis matrix is updated to %$\bW_t$,  %, $\{\bh_i\}_{i\in[t]}$ and $\{\br_i\}_{i\in[t]}$
\begin{align}
\bW_t = \argmin_{\bW\in\calC} \tilf_t(\bW),\label{eq:min_W}
\end{align}
where 
\begin{equation}
\tilf_t(\bW)\defeq\frac{1}{t}\sum_{i=1}^t \frac{1}{2}\norm{\bv_i-\bW\bh_i-\br_i}_2^2+\lambda\norm{\br_i}_1. \label{eq:tilde_f_t}
\end{equation}
We note that \eqref{eq:min_W} can be rewritten as
\begin{align}
\bW_t = \argmin_{\bW\in\calC} \frac{1}{2}\tr\left(\bW^T\bW\bA_t\right) - \tr\left(\bW^T\bB_t\right), \label{eq:min_W_tr}
\end{align}
where $\bA_t\defeq1/t\sum_{i=1}^t \bh_i\bh_i^T$ and $\bB_t \defeq 1/t\sum_{i=1}^t (\bv_i-\br_i)\bh_i^T$ are the sufficient statistics. From \eqref{eq:min_W_tr}, we observe that our algorithm has a storage complexity independent of $t$ since only $\bA_t$, $\bB_t$ and $\bW_t$ need to be stored and updated.
% and ii) $\bW_t$ is a function of only $\bA_t$ and $\bB_t$ if we assume \eqref{eq:min_W_tr} has a unique minimizer. 
%As shown below, the dictionary update step in \eqref{eq:min_W} only relies on two sufficient statistics %defined as 
%Thus, we can achieve constant storage complexity throughout the online algorithm. 



%We have two sub-problems.\\
%First, for fixed $\bv$ and $\bW$, we need to solve
%\begin{align*}
%%&\min\;\; \frac{1}{2}\norm{\bv_t-\bW_{t-1}\bh-\br}_2^2 + \lambda\norm{\br}_1\\
%&\min\;\; \frac{1}{2}\norm{\bv-\bW\bh-\br}_2^2 + \lambda\norm{\br}_1\\
%&\st\;\; \bh\geq 0,\norm{\br}_\infty\le  M \numberthis\label{eq:min_hr} 
%\end{align*}
%%We restrain the outlier $\br$ to be bounded because %also provide corresponding algorithms. 
%Second, for fixed $\{\bh_i\}_{i\in[t]}$ and $\{\br_i\}_{i\in[t]}$, we need to solve
%\begin{equation}
%\min_{\bW\in\calC}\frac{1}{t}\sum_{i=1}^t\frac{1}{2}\norm{\bv_i-\bW\bh_i-\br_i}_2^2.\label{eq:min_W}
%\end{equation}
%where $\calC \defeq \left\{\bW\in\bbR_+^{F\times K}\,|\,\norm{\bW_i}_2\le 1, \forall\,i\in[K]\right\}$ is a closed convex set. This normalization constraint is a convention in the literature of NMF to prevent entries of $\bW$ to be arbitrarily large. 

%For outliers with or without nonnegativity constraints, 
To solve \eqref{eq:min_hr} and \eqref{eq:min_W_tr} ({\em at a fixed time instant $t$}), we propose two solvers based on PGD and ADMM respectively. For ease of reference, we refer to the former algorithm as OPGD and the latter as OADMM.
%The reasons that we propose two algorithms are illustrated as follows. 
We now explain the motivations behind proposing these two solvers. Since both \eqref{eq:min_hr} and \eqref{eq:min_W_tr} are constrained optimization problems, the most straightforward solver would be based on PGD. Although such a solver has a linear computational complexity per iteration, it typically needs a large number of iterations to converge. Moreover, it is also easily trapped in bad local minima \cite{Sun_14}. Thus, it would be meaningful to contrast its performance with a solver with very different properties. This leads us to propose another solver based on ADMM. Such a solver has a higher computational complexity per iteration but typically needs fewer number of iterations to converge \cite{Xu_12}. It is also less susceptible to bad local minima since it solves optimization problems in the dual space. In Section~\ref{sec:numericals}, we will show that the practical performances of these two solvers are comparable despite the different properties they possess. Thus either solver can be used for most practical purposes. 
%In fact, the solvers based on PGD \cite{Lin_07a} and ADMM \cite{Xu_12,Sun_14} have been applied to the canonical NMF problem. 
%However, these two classes of algorithms have different properties. 
%can be regarded as a special case of a more general class of algorithms, namely the majorization-minimization (MM) algorithms. MM has been widely applied in solving NMF problems. 
%Specifically, PGD-based algorithms have linear computational complexity per iteration %However, as pointed in \cite{Sun_14}, MM-based algorithms also 
%but typically need a large number of iterations to converge. %suffers from slow convergence (in terms of number in) 
%Moreover, they are easily trapped in bad local minima \cite{Sun_14}. %only asymptotic convergence to zeros. 
%Recently ADMM-based algorithms have been shown to overcome the drawbacks of those based on MM and obtain promising performance in solving the NMF problems \cite{Sun_14,Xu_12}. However, 
%In contrast, ADMM-based algorithms have higher computational complexity per iteration but typically need fewer number of iterations to converge. They are also less susceptible to bad local minima since they solve optimization problems in the dual space. 
%and are more complicated to implement. 
%Since these two algorithms are complementary, we propose both in order for practitioners to choose based on the real applications. The experimental performances of the two algorithms
% together with that of other similar matrix factorization algorithms, 
%are compared in Section~\ref{sec:numericals}. 

In the sequel, we omit the time subscript $t$ to keep notations uncluttered. %We use $k\in\bbN$ to index the iterations. 
In the iterations, the updated value of a variable is denoted with the superscript `$+$'. 
%Also, the (unique) optimal solution of an optimization problem is denoted with the superscript $*$.
Pseudo-codes of the entire algorithm (for $N$ data samples) are provided in Algorithm~\ref{algo:generic}. %Note that $N$ can be both finite or infinite. 

%\subsection{Real Outliers}
\subsection{Online Algorithm Based on PGD (OPGD)}\label{sec:OPGD}
%We can adapt the batch algorithm derived in \cite{Zhang_11} to this case. The update rules are
%We first derive an algorithm for \eqref{eq:min_hr}.  
\subsubsection{PGD solver for \eqref{eq:min_hr}} \label{sec:algoMM_minhr}
%Based on the principle of {\em block coordinate descent}, for a fixed $\bW$, we solve \eqref{eq:min_hr} by alternating between solving the following two problems
%\begin{align}
%&\hspace{-.3cm}\min_\bh \;  q(\bh) + I_+(\bh), \,\mbox{where}\, q(\bh) = \frac{1}{2}\norm{\bv-\bW\bh-\br}_2^2, \label{eq:min_h} \\
%%\min_{\br\in\calR} &q_2(\bh), \;\mbox{where}\;  \frac{1}{2}\norm{\bv-\bW\bh-\br}_2^2 + \lambda\norm{\br}_1. \label{eq:min_r}
% &\hspace{-.3cm}\min_\br \; \frac{1}{2}\norm{\bv-\bW\bh-\br}_2^2 + \tilq(\br), \nn\\
%&\hspace{2.5cm}\mbox{where}\; \tilq(\br) = \lambda\norm{\br}_1 + I_\calR(\br). \label{eq:min_r}
%\end{align}

%We recognize \eqref{eq:min_h} as a quadratic program with nonnegativity constraint. Therefore, using Lemma~\ref{lem:nonneg_qp}, %\cite[Theorem 1]{Fei_03}, 
%the multiplicative update for $\bh$ is
%\begin{equation}
%\bh^+ := \bh \cdot \frac{\bW^T(\bv-\br)+\left\vert\bW^T(\bv-\br)\right\vert}{2\bW^T\bW\bh}.
%\end{equation}

%Since $q$ is convex and differentiable, $\nabla q$ is Lipschitz with a constant $L\defeq\normt{\bW^T\bW}_2 = \normt{\bW}_2^2$ and $I_+$ is proper, closed and convex, the PGD algorithm can be readily applied \cite{Parikh_14}. 
%Fix a $\bh\ge 0$, we first construct a quadratic majorant,\footnote{For a function $g$ with domain $\calG$, its majorant at $\kappa\in\calG$, $\tilg$ is the function that satisfies i) $\tilg\ge g$ on $\calG$ and ii) $\tilg(\kappa) = g(\kappa)$.} $Q_\eta(\cdot|\bh)$ for $q$ at $\bh$ %as follows
%\begin{equation}
%Q_\eta(\bh'\vert \bh) = q(\bh) + \lrangle{\nabla q(\bh)}{\bh'-\bh} + \frac{1}{2\eta} \norm{\bh'-\bh}_2^2,
%\end{equation}
%where $\nabla q(\bh) = \bW^T(\bW\bh+\br-\bv)$ and $\eta\in(0,1/L]$.\footnote{At time $t$, the value of $L$ is computed based on $\bW_{t-1}$, i.e., $L_t = \norm{\bW_{t-1}}_2^2$.} Minimizing $Q_\eta(\bh'|\bh) + I_+(\bh')$ over $\bh'\ge 0$, we have %the 	
%%Alternatively, denote $\pi(\bh) = 1/2\norm{\bv-\bW\bh-\br}_2^2$,
%%$\tilell_\bh$ as the mapping $\bh\mapsto\tilell(\bv,\bW,\bh,\br)$, 
%%update as %based on proximal gradient method is
%\begin{equation}
%\bh^+ := \calP_+(\bh - \eta\nabla q(\bh)).
%\end{equation}
%For ease of parameter tuning in Section~\ref{sec:numericals}, we rewrite $\eta = \barkappa/L$ ($0<\barkappa\le 1$). Furthermore, we fix $\eta$ (or $\barkappa$) throughout all iterations. Due to the Lipschitz continuity of $\nabla q$, the algorithm will converge with rate $O(1/k)$,\footnote{By this we mean $q(\bh^k)-q^* = O(1/k)$, where $\bh^k$ is the $k$-th iterate and $q^*$ is the optimal value of \eqref{eq:min_h}.} where $k$ denotes the number of iterations \cite{Parikh_14}. 
%Since $\nabla q(\bh)$ is Lipschitz with a constant $\norm{\bW^T\bW}_2$, we fix the step size $\lambda$ as a constant in $(0,1/\alpha]$ throughout all iterations. 


For a fixed $\bW$, we solve \eqref{eq:min_hr} by alternating between the following two steps
\begin{align}
&\bh^+ := \argmin_{\bh'\ge 0} Q_\eta(\bh'\vert \bh),\label{eq:min_h}\\
&\br^+ := \argmin_{\br'\in\calR} \; \frac{1}{2}\norm{\bv-\bW\bh^+-\br'}_2^2 + \lambda\norm{\br'}_1,\label{eq:min_r}
\end{align}
where\footnote{At time $t$, the value of $L$ is computed based on $\bW_{t-1}$, i.e., $L_t \defeq \norm{\bW_{t-1}}_2^2$.}  
\begin{equation}
Q_\eta(\bh'\vert \bh) \defeq q(\bh) + \lrangle{\nabla q(\bh)}{\bh'-\bh} + \frac{1}{2\eta} \norm{\bh'-\bh}_2^2,\nn
\end{equation}
$q(\bh) \defeq \frac{1}{2}\norm{\bv-\bW\bh-\br}_2^2$ and  $\eta\in\left(0,1/L\right]$ with $L\defeq \norm{\bW}_2^2$. The steps \eqref{eq:min_h} and \eqref{eq:min_r} can be interpreted based on the framework of {\em block MM} \cite{Raza_13,Hong_16}. Specifically, it is easy to verify that the steps \eqref{eq:min_h} and \eqref{eq:min_r} amount to finding the (unique) minimizers of the majorant functions\footnote{For a function $g$ with domain $\calG$, its majorant at $\kappa\in\calG$, $\tilg$ is the function that satisfies i) $\tilg\ge g$ on $\calG$ and ii) $\tilg(\kappa) = g(\kappa)$.} of $\bh'\mapsto\tilell(\bv,\bW,\bh',\br)$ at $\bh$ and $\br'\mapsto\tilell(\bv,\bW,\bh^+,\br')$ at $\br$ respectively. Therefore, the convergence analysis in \cite{Raza_13} guarantees that such alternating minimization procedure converges to a global optimum of \eqref{eq:min_hr}. 

In addition, we notice that the minimizations in both \eqref{eq:min_h} and \eqref{eq:min_r} have closed-form solutions. For \eqref{eq:min_h}, the solution is given by the PGD update step (with constant step size)
\begin{equation}
\bh^+ := \calP_+(\bh - \eta\nabla q(\bh)).
\end{equation}
For ease of parameter tuning, we rewrite $\eta = \barkappa/L$ ($0<\barkappa\le 1$). Furthermore, we fix $\eta$ (or $\barkappa$) throughout all iterations.

% Unlike   \eqref{eq:min_h}, we can derive a closed-form solution for \eqref{eq:min_r}. We recognize~\eqref{eq:min_r} as a proximal minimization\footnote{For a function $g$ with domain $\calG\subseteq \bbR^n$ and $x\in\bbR^n$, $\prox_g(x) = \argmin_{y\in\calG} 1/2\norm{x-y}_2^2 + g(y)$.} problem
%\begin{equation}
%\br^* := \prox_{\tilq} (\bv-\bW\bh). \label{eq:prox_r}
%\end{equation}
%In other words, we need to find the proximal operator of the $\ell_1$ norm with box constraints. 
For \eqref{eq:min_r}, if $M=\infty$, the solution is precisely given by  
\begin{equation}
\br^+ := \calS_{\lambda} (\bv-\bW\bh^+),
\end{equation}
where $\calS_{\lambda}$ is the (elementwise) soft-thresholding operator threshold $\lambda$. Otherwise, when $M$ is finite, using \cite[Lemma~5]{ZhangBox_15} (see Lemma~\ref{lem:L1_box}), we have %see \eqref{eq:prox_r} can be implemented via the operator
%obtain an operator $\widetilde{\calS}_{\lambda,M}$ with similar form to $\calS_\lambda$ %this operator is given in a similar form as the well-known soft-thresholding operator, which acts as the proximal operator for the unconstrained $\ell_1$ norm
\begin{equation}
\br^+ := \widetilde{\calS}_{\lambda,M} (\bv-\bW\bh^+),
\end{equation}
where for any $\bx\in\bbR^F$ and $i\in[F]$,
\begin{equation*}
\left(\widetilde{\calS}_{\lambda,M}(\bx)\right)_i :=\begin{cases}
0, &|x_i|<\lambda\\
x_i - \sgn(x_i)\lambda, & \lambda \le |x_i| \le \lambda+M\\
\sgn(x_i)M, &|x_i| > \lambda+M
\end{cases}.
\end{equation*}
%where $\bx\in\bbR^F$ and $i\in[F]$. Therefore, we have
%\begin{equation}
%\br^* := \widetilde{\calS}_{\lambda,M} (\bv-\bW\bh).
%\end{equation}
%where $\widetilde{\calS}_{\lambda,M}$ is applied elementwise to $\bv-\bW\bh$ and for a scalar $x$, 


%In the case where it is hard to find a suitable finite $M$, we can simply set $M=\infty$. In such case, we have
%where $\calS_{\lambda}$ is the well-known soft-thresholding operator with threshold $\lambda$ \cite{Parikh_14} applied elementwise to $\br$.
%For a scalar $x$, $\widetilde{\calS}(x) = \Pi_{[-M,M]}\left(\calS_\lambda(x)\right)$, where $\Pi_{[-M,M]}$ is the projection operator onto interval $[-M,M]$ and $\calS_\lambda(x)$ is the soft-thresholding operator with threshold $\lambda$.

\subsubsection{PGD solver for \eqref{eq:min_W_tr}} \label{sec:algoMM_minW}
Similar to the procedure for solving \eqref{eq:min_h}, we first rewrite \eqref{eq:min_W_tr} as 
\begin{align}
&\min_\bW p_t(\bW) + I_{\calC}(\bW), \mbox{where}\nn\\
& \qquad p_t(\bW) = \frac{1}{2}\tr\left(\bW^T\bW\bA_t\right) - \tr\left(\bW^T\bB_t\right).
\end{align}
%where $I_{\calC}$ is the indicator function of set $\calC$ and $\bV_t \defeq [\bv_1 \ldots \bv_t]$ (similar definitions apply to $\bH_t$ and $\bR_t$). For convenience we denote $h_t(\bW) \defeq 1/2\norm{\bV_t - \bW\bH_t - \bR_t}_F^2$. Since $h_t(\bW)$ is convex and strongly smooth %(\wrt $\bW$) 
%and the second term $I_{\calC}(\bW)$ is proper convex, we make use of the proximal gradient method \cite{Parikh_14} (a subclass of the MM algorithms) to solve \eqref{eq:min_W2}. 
First, it is easy to see $p_t$ is convex and differentiable, $\nabla p_t$ is Lipschitz with constant $\tilL_t\defeq\norm{\bA_t}_F$. Thus, we can construct a majorant function $P_t(\bW'|\bW)$ for $p_t(\bW')$ at $\bW\in\calC$
\begin{align}
&P_t(\bW'|\bW) = p_t(\bW) + \lrangle{\nabla p_t(\bW)}{\bW'-\bW}\nn \\
&\hspace{4cm}   + \frac{1}{2\tmeta_t}\norm{\bW'-\bW}_F^2,
\end{align}
where $\nabla p_t(\bW) = \bW\bA_t-\bB_t$ and $\tmeta_t\in(0,1/\tilL_t]$. 
%Fix a point $\bW\in\calC$, we construct an upperbound, $H_t(\bW')$ for the objective function in \eqref{eq:min_W2} as
%\begin{equation}
%H_t(\bW') \defeq h_t(\bW) + \lrangle{\nabla h_t(\bW)}{\bW'-\bW} + \frac{1}{2\eta_t}\norm{\bW'-\bW}_F^2 + I_\calC(\bW'),
%\end{equation}
%where $\eta_t\in(0,1/L_t]$. 
Minimizing $P_t(\bW'|\bW)+I_\calC(\bW')$ over $\bW'\in\calC$, we have
\begin{align}
\bW^+ &:= \argmin_{\bW'\in\calC} \norm{\bW'-(\bW-\tmeta_t\nabla p_t(\bW))}_F^2\\
&:= \calP_{\calC} (\bW-\tmeta_t\nabla p_t(\bW)). \label{eq:proj_W}
\end{align}
By Lemma~\ref{lem:proj_nonnegL2}, the projection step in \eqref{eq:proj_W} is given by  
\begin{equation}
\bW^+_{:j} := \frac{\calP_+(\bW-\tmeta_t\nabla p_t(\bW))_{:j}}{\max\{1,\norm{\calP_+(\bW-\tmeta_t\nabla p_t(\bW))_{:j}}_2\}}, \;\forall\,j\in [K].
\end{equation}
Again, for each iteration given in \eqref{eq:proj_W}, we use the same step size $\tmeta_t = \tkappa_t/L$ where $0<\tkappa_t\le 1$.
%$\eta_t$ is fixed though all iterations due to Lipschitz property of $\nabla p_t$.
%where $\bW_j$ is the $j$-th column of $\bW$.
%Since the online setting prohibits storing $\bV_t$, $\bH_t$ and $\bR_t$, we express $\nabla h_t(\bW)$ and $L_t$ in terms of sufficient statistics $\bA_t\defeq\sum_{i=1}^t \bh_i\bh_i^T$ and $\bB_t = \sum_{i=1}^t (\bv_i-\br_i)\bh_i^T$
%\begin{align*}
%\nabla h_t(\bW) = \bW\bA_t - \bB_t \qquad \mathrm{and}\qquad L_t = \norm{\bA_t}_F.
%\end{align*}

\begin{remark}\label{rmk:acc_grad}
In the literature\cite{Tseng_08,Nest_13}, the accelerated proximal gradient descent (APGD) method  has been proposed to accelerate the canonical PGD method. With an additional extrapolation step in each iteration, the convergence rate can be improved from $O(1/k)$ to $O(1/k^2)$, where $k$ denotes the number of iterations. However, since in our implementations, both \eqref{eq:min_hr} and \eqref{eq:min_W_tr} were only solved to a prescribed accuracy (see Section~\ref{sec:stop_crit}), we observed  no significant reduction in running times on the real tasks. Thus for simplicity, we only employ the canonical PGD method. 
\end{remark}

\subsection{Online Algorithm Based on ADMM (OADMM)}\label{sec:ADMM}
Below we only present the update rules derived via ADMM. The detailed derivation steps are shown in Section~\ref{sec:deriv_ADMM}. %the supplemental material. %Section~\ref{sec:deriv_ADMM}.
\subsubsection{ADMM solver for \eqref{eq:min_hr}} \label{sec:algoADMM_minhr}
First we reformulate \eqref{eq:min_hr} as
\begin{align}
&\min_{\bh,\br}\;\; \frac{1}{2}\norm{\bv-\bW\bh-\br}_2^2 + \lambda\norm{\br}_1 + I_+(\bu) + I_\calR(\bq)\nn\\
&\st \;\; \bh = \bu, \br = \bq %, \bu\ge 0, \norm{\bq}_\infty\le M
\end{align}
Thus the augmented Lagrangian is 
\begin{align}
&\calL(\bh,\br,\bu,\bq,\balpha,\bbeta) = \frac{1}{2}\norm{\bv-\bW\bh-\br}_2^2 + \lambda\norm{\br}_1  \nn\\
&+ I_+(\bu) + I_\calR(\bq) +\balpha^T(\bh-\bu) + \bbeta^T(\br-\bq) \nn\\
&+\frac{\rho_1}{2}\norm{\bh-\bu}^2_2 + \frac{\rho_2}{2}\norm{\br-\bq}_2^2, 
\end{align}
where $\balpha$, $\bbeta$ are dual variables and $\rho_1$, $\rho_2$ are positive penalty parameters. 
Then we sequentially update each of $\bh$, $\br$, $\bu$, $\bq$, $\balpha$ and $\bbeta$ while keeping other variables fixed 
\begin{align}
\bh^+ &:= (\bW^T\bW+\rho_1\bI)^{-1}\left(\bW^T(\bv-\br) + \rho_1\bu-\balpha\right)\label{eq:ADMMupd_h}\\
\br^+ &:= \calS_\lambda(\rho_2\bq+\bv-\bbeta-\bW\bh)/(1+\rho_2)\\
\bu^+ &:= \calP_+\left(\bh^+ +\balpha/\rho_1\right)\\
\bq^+ &:= \calP_\calR\left(\br^+ +\bbeta/\rho_2\right)\\
\balpha^+ &:= \balpha + \rho_1(\bh^+-\bu^+)\\
\bbeta^+ &:= \bbeta + \rho_2(\br^+-\bq^+).
\end{align}
%where for a vector $\bx$, $\bx^+$ denotes its updated value. %and $\calQ = \{\bq\in\bbR^F\,|\,\norm{\bq}_\infty\le M\}$. %, $\calS(\bx)$ denotes the result of applying the soft-thresholding operator $\calS$ elementwise to $\bx$ and the $i$-th  element of $(\bx)_+$ equals $\max\{\bx_i,0\}$.

\subsubsection{ADMM solver for \eqref{eq:min_W_tr}} \label{sec:algoADMM_minW}
%Similarly we derive an algorithm for \eqref{eq:min_W_tr}. 
Again, we rewrite \eqref{eq:min_W_tr} as
\begin{align}
&\min \;\; \frac{1}{2}\tr\left(\bW^T\bW\bA_t\right) - \tr\left(\bW^T\bB_t\right) + I_\calC(\bQ)\nn\\
&\st \;\; \bW=\bQ %, \bQ\in \calC
\end{align}
%where $\bV_t \defeq [\bv_1 \ldots \bv_t]$ and similar definitions apply to $\bH_t$ and $\bR_t$. \\
The augmented Lagrangian in this case is
\begin{align}
&\calL(\bW,\bQ,\bD) = \frac{1}{2}\tr\left(\bW^T\bW\bA_t\right) - \tr\left(\bW^T\bB_t\right) + I_\calC(\bQ)\nn\\
&\hspace{2cm}+ \lrangle{\bD}{\bW-\bQ} + \frac{\rho_3}{2}\norm{\bW-\bQ}_F^2,
\end{align}
where $\bD$ is the dual variable and $\rho_3$ is a positive penalty parameter. 
Minimizing $\bW$, $\bQ$ and $\bD$ sequentially yields %Therefore the update rule is
\begin{align}
\bW^+ %&:=& \left((\bV_t-\bR_t)\bH_t^T-\bD+\rho_3\bQ\right)\left(\bH_t\bH_t^T+\rho_3\bI\right)^{-1} \nn\\
&:= \left(\bB_t-\bD+\rho_3\bQ\right)\left(\bA_t+\rho_3\bI\right)^{-1}\label{eq:ADMMupd_W}\\
\bQ^+ &:=  \calP_\calC\left(\bW^++\bD/\rho_3\right)\label{eq:ADMMupd_Q}\\
\bD^+ &:=  \bD + \rho_3\left(\bW^+-\bQ^+\right).
\end{align}
\begin{remark}\label{rmk:algo}
A few comments are now in order: First, although  algorithms based on both PGD and ADMM exist in the literature for the canonical NMF problem  (see for example, \cite{Lin_07a,Sun_14,Xu_12}), our problem setting is different from those in the previous works. Specifically, our problem explicitly models the outlier vector $\br$ and the constraint set on $\bW$ is more complicated than the non-negative orthant $\bbR_+^{F\times K}$. % and $\br$, both solvers are different from that. 
Moreover, for the algorithm based on PGD, we use a fixed step size though all the iterations. In contrast, the usual projected gradient method applied to NMF \cite{Lin_07a} involves  computationally intensive Armijo-type line searches for the step sizes.
Second, although updates \eqref{eq:ADMMupd_h} and \eqref{eq:ADMMupd_W} involve matrix inversions, the operations do not incur high computational complexity since both matrices to be inverted have sizes $K\times K$ where $K \ll F$. % which is typically computationally expensive (). However, , so the inversion operation only has complexity $O(K^3)$. 
Third, the proposed two solvers can be easily extended to solve the batch NMF problem with outliers. Thus, we have also effectively proposed {\em two new solvers} for the (batch) robust NMF problem. These extensions are detailed in Section~\ref{sec:BRNMF_algo}. % in the supplemental material. 
In the sequel we term these two batch algorithms as BPGD and BADMM respectively. 
Finally, the stopping criteria of both OPGD and OADMM for solving \eqref{eq:min_hr} and \eqref{eq:min_W_tr} will be described in Section~\ref{sec:stop_crit}. 
\end{remark}


\begin{algorithm}[t]
\caption{Online NMF with outliers (ONMFO)}\label{algo:generic}
\begin{algorithmic} 
\State {\bf Input}: Data samples $\{\bv_i\}_{i\in[N]}$, penalty parameter $\lambda$, initial dictionary matrix $\bW_0$
\State {\bf Initialize} sufficient statistics: $\bA_0 := \vecz$, $\bB_0 := \vecz$
\State {\bf for} $t$ = 1 to $N$ {\bf do}
\State \quad 1) Acquire a data sample $\bv_t$.
\State \quad 2) Learn the coefficient vector $\bh_t$ and the outlier vector $\br_t$ based on $\bW_{t-1}$, % and $\bv_t$, 
using the solvers based on PGD or ADMM  (detailed  in Sections~\ref{sec:algoMM_minhr} and~\ref{sec:algoADMM_minhr})
%iteratively, using \eqref{eq:update_h} and \eqref{eq:update_r} %using Algorithm~\ref{algo:learn_hr}
\begin{align}
&\hspace{-.3cm}\left(\bh_t, \br_t\right) := \argmin_{\bh\geq 0,\br\in\calR} \tilell(\bv_t,\bW_{t-1}, \bh, \br).\label{eq:learn_hr}
\end{align} 
\State \quad 3) Update the sufficient statistics 
\begin{align*}
\bA_t &:= {1}/{t}\left\{(t-1)\bA_{t-1} + \bh_t\bh_t^T\right\}, \\
\bB_t &:= 1/t\left\{(t-1)\bB_{t-1} + (\bv_t-\br_t)\bh_t^T\right\}.
%\bC_t &:= \bC_{t-1} + \br_t\bh_t^\top.
\end{align*}
\State \quad 4) Learn the dictionary matrix $\bW_t$ based on $\bA_t$ and $\bB_t$, using the solvers based on PGD or ADMM  (detailed in Section~\ref{sec:algoMM_minW} and \ref{sec:algoADMM_minW})
%using Algorithm~\ref{algo:update_W}
\begin{equation}
\bW_t := \argmin_{\bW\in\calC}\frac{1}{2}\tr\left(\bW^T\bW\bA_t\right) - \tr\left(\bW^T\bB_t\right) \label{eq:dict_upd}
%\bW_t := \argmin_{\bW\in\calC}\frac{1}{t}\sum_{i=1}^t\frac{1}{2}\norm{\bv_i-\bW\bh_i-\br_i}_2^2+\lambda\norm{\br_i}_1
%\bW_t = \argmin_{\bW\geq 0} \frac{1}{2}\norm{\bV_t-\bW\bH_t-\bR_t}_F^2+\lambda\norm{\bR_t}_{1,1}.
\end{equation}
\State {\bf end for}
\State {\bf Output}: Final dictionary matrix $\bW_N$% (\textcolor{red}{what is $N$ in the subscript here?})
\end{algorithmic}
\end{algorithm}



\section{Convergence Analyses}\label{sec:conv_analysis}
In this section, we analyze the convergence of both the sequence of objective values $\{f(\bW_t)\}_{t\ge 1}$ (Theorem~\ref{thm:as_conv}) and the sequence of dictionary matrices $\{\bW_t\}_{t\ge 1}$ (Theorem~\ref{thm:local_min}) produced by Algorithm~\ref{algo:generic}. It is worth noticing that the analyses are independent of the specific solvers used in steps 2) and 4) in Algorithm~\ref{algo:generic}. 
%Different solvers may affect on the convergence rates towards the global optima in the convex programs \eqref{eq:min_hr} and \eqref{eq:min_W_tr}. 
Indeed, in our analyses, we only leverage the fact that both \eqref{eq:min_hr} and \eqref{eq:min_W_tr} can be exactly solved. 
%This is because our problem is nonconvex in nature, so we are unable to show any results on the convergence rates of the sequences generated from PGD and ADMM. 

\subsection{Preliminaries}\label{sec:prelim}
First, given any $\bh'\ge 0$ and $\br'\in\calR$, we observe that  $(\bv,\bW)\mapsto\tilell(\bv,\bW, \bh', \br')$ and $\tilf_t$ serve as upper-bound functions for $\ell$ (on $\bbR_+^F\times\calC$) and $f_t$ (on $\calC$) respectively. %Here $\{\bh_i\}_{i\in[t]}$ and $\{\br_i\}_{i\in[t]}$ are past statistics learned in Algorithm~\ref{algo:generic}.   
%This is because given $\bW$, for all $\bh\ge 0$ and $\br\in\calR$, we have $\tilell(\bv,\bW, \bh, \br) \ge \ell(\bv,\bW)$ and $\tilell(\bv,\bW, \bh^*, \br^*) = \ell(\bv,\bW)$, %In particular, $\tilell(\bv,\bW, \bh, \br)$ where 
Denote $(\bh^*,\br^*)$ as an optimal solution of \eqref{eq:ith_loss}. (Note that $(\bh^*,\br^*)$ is a function of $\bv$ and $\bW$ and always exists since $\tilell(\bv',\bW', \bh, \br)$ is closed and coercive on $\bbR_+^K\times \calR$ for any $\bv'\in\bbR_+^F$ and $\bW'\in\calC$.) Clearly we have $\tilell(\bv,\bW, \bh^*, \br^*) = \ell(\bv,\bW)$, for any $(\bv,\bW)\in\bbR_+^F\times\calC$. %Moreover, the dictionary update step \eqref{eq:dict_upd} can equivalently written as $\bW_t := \min_{\bW\in\calC} \tilf_t(\bW)$.


Next, we notice that $\ell$ can be equivalently defined as 
\begin{equation}
\ell(\bv,\bW) = \min_{(\bh,\br)\in\calH\times\calR} %\min_{\bh\in\calH,\br\in\calR} 
\tilell(\bv,\bW, \bh, \br), \label{eq:new_ithloss}
\end{equation}
where $\calH$ is a compact and convex set in $\bbR^K_+$. This is because $\bh^*$ in \eqref{eq:ith_loss} is bounded, due to the boundedness of $\bv_i$ and $\bW$. Thus it suffices to consider minimizing $\bh$ over a compact set in $\bbR^K_+$. We can further choose $\calH$ to be convex. If $M=\infty$, we can similarly show $\br^*$ is bounded thus still consider minimizing $\br$ over a compact and convex set $\calR$ in $\bbR^F$. 

We make three assumptions for our  subsequent analyses.
\begin{assump}\quad
\begin{enumerate}
\item The data generation distribution $\bbP$ has a compact support set $\calV$. \label{assum:comp_supp}
\item For all $(\bv,\bW)\in\calV\times\calC$, $\tilell(\bv,\bW, \bh, \br)$ is {\em jointly} $m_1$-strongly convex in $(\bh,\br)$ for some constant $m_1>0$.\label{assum:sc_hr}
\item For all $t\ge 1$, $\tilf_t(\bW)$ is $m_2$-strongly convex on $\calC$ for some constant $m_2>0$. \label{assum:sc_W}
%Each $\bW_t$ has full column rank, $\forall\,t\ge 0$. \label{assum:full_rank}
\end{enumerate}
\end{assump}
\begin{remark}\label{rmk:assump}
All of the assumptions above are made to simplify the convergence analyses. 
%Among them, Assumption~\ref{assum:sc_W} can be removed if we choose the majorant $$
%These assumptions are reasonable in the following sense. 
Among them, Assumption~\ref{assum:comp_supp} naturally holds for real data, which are always bounded. %Assumptions~\ref{assum:sc_hr} and \ref{assum:sc_W} are mainly for the convenience of analysis. 
Assumptions~\ref{assum:sc_hr} and \ref{assum:sc_W} play roles in proving Lemma~\ref{lem:regularity} and \ref{lem:rate_W} respectively. Apart from this, they have no effects in proving our main theorems (i.e., Theorem \ref{thm:as_conv} and \ref{thm:local_min}). These two assumptions hold by simply adding strongly convex regularizers to $\tilell(\bv,\bW, \bh, \br)$ or $\tilf_t(\bW)$. For example, we can add Tikhonov regularizer $(\nu_1\norm{\bh}_2^2 + \nu_2\norm{\br}_2^2)/2$ (for some positive $\nu_1,\nu_2$) to $\tilell(\bv,\bW, \bh, \br)$, then in such case $m_1 = \min(\nu_1,\nu_2)$. Adding such regularizers can be regarded as a way to promote smoothness and avoid over-fitting on $\bW$, $\bh$ and $\br$ (see Section~\ref{sec:extension}). Moreover, including the regularizers in $\tilell(\bv,\bW, \bh, \br)$ or $\tilf_t(\bW)$ will not alter any steps in our analyses. 
In terms of our algorithms, for small regularization weights (e.g., $\nu_1,\nu_2\ll 1$), such regularizers will only slightly alter steps \eqref{eq:learn_hr} and \eqref{eq:dict_upd} in Algorithm~\ref{algo:generic}. For example, in \eqref{eq:learn_hr}, $\tilell(\bv_t,\bW_{t-1}, \bh, \br)$ will become
\begin{align}
&\tilell'(\bv_t,\bW_{t-1}, \bh, \br) \defeq \frac{1}{2}\norm{\bv-\bW\bh-\br}_2^2 +\frac{\nu_1}{2}\norm{\bh}_2^2 \nn\\
&\hspace{4.5cm}+\lambda\norm{\br}_1 + \frac{\nu_2}{2}\norm{\br}_2^2.
\end{align}
Thus for simplicity we omit such regularizers in the algorithm. However, both of our solvers (OPGD and OADMM) can be straightforwardly adapted to the regularized case. 
%Moreover, besides adding the Tikhonov regularizer,  Assumption~\ref{assum:sc_W} also holds if we choose the upper-bound function $\tilf_t$ in a slightly different way. See Section~\ref{sec:rmv_scW} for details. 
\end{remark}

Finally, we define the notion of {\em stationary point} of a constrained optimization problem. %, an important concept used in the statement of our main results.
\begin{definition}\label{def:stat_pt}
Given a real Hilbert space $\calX$, a differentiable function $g:\calX\to\bbR$  and a set $\calK\subseteq\calX$, $x_0\in\calK$ is a stationary point of $\min_{x\in\calK}g(x)$ if $\lrangle{\nabla g(x_0)}{x-x_0}\ge 0$, for all $x\in\calK$.
\end{definition}

\subsection{Main Results and Key Lemmas}
In this section we present our main  results (Theorem~\ref{thm:as_conv} and \ref{thm:local_min}) 
and key lemmas to prove these results (Lemma~\ref{lem:regularity} and \ref{lem:rate_W}). We only show proof sketches here. Detailed proofs are deferred to %the supplemental material. 
Section~\ref{sec:regularity} to \ref{sec:Lips_b} in the supplemental material. 

%Our main results consist of two theorems. Theorem~\ref{thm:as_conv} states that the sequence of objective values $\{f(\bW_t)\}_{t\ge 1}$ converges almost surely (a.s.). 
% convergence of under Algorithm~\ref{algo:generic}. 
%However, it does not indicate any convergence properties of $\{\bW_t\}_{t\ge 1}$. These properties are stated in Theorem~\ref{thm:local_min}. Specifically, we show the sequence of dictionaries $\{\bW_t\}_{t\ge 1}$ converges to the set of  stationary points of \eqref{eq:exp_loss} almost surely (a.s.). 

\begin{theorem}[Almost sure convergence of the sequence of objective values]\label{thm:as_conv}
In Algorithm~\ref{algo:generic}, the nonnegative stochastic process $\{\tilf_t(\bW_t)\}_{t\ge 1}$ converges a.s.. Furthermore, $\{f(\bW_t)\}_{t\ge 1}$ converges to the same almost sure limit as $\{\tilf_t(\bW_t)\}_{t\ge 1}$. 
\end{theorem}
\begin{psketch}
The proof proceeds in two major steps. First, making use of the quasi-martingale convergence theorem \cite[Theorem 9.4 \& Proposition 9.5]{Met_82} (see Lemma~\ref{lem:quasi_conv}), we prove $\{\tilf_t(\bW_t)\}_{t\ge 1}$ converges a.s.\ by showing the series of the expected positive variations of this process converges. A key step in proving the convergence of this series  is to bound each summand of the series by Donsker's theorem (see Lemma~\ref{lem:Donsker}). To invoke this theorem, we show the class of measurable functions $\{\ell(\cdot,\bW):\bW\in\calC\}$ is $\bbP$-Donsker \cite{Vaart_00} using the Lipschitz continuity of $\ell(\bv,\cdot)$ on $\calC$ (see Lemma~\ref{lem:regularity}) and \cite[Example 19.7]{Vaart_00} (see Lemma~\ref{lem:suff_Donsker}). %This result enables us to invoke Donsker's theorem (see Lemma~\ref{lem:Donsker}) to  
Second, we show 
\begin{equation}
f_t(\bW_t)-\tilf_t(\bW_t)\convas 0.\label{eq:as_conv_diff}
\end{equation}
by showing $\sum_{t=1}^\infty \frac{\tilf_t(\bW_t)-f_t(\bW_t)}{t+1}$ converges a.s.. Define $b_t \defeq \tilf_t(\bW_t)-f_t(\bW_t)$. Using the Lipschitz continuity of both $\tilf_t$ and $f_t$ on $\calC$ and Lemma~\ref{lem:rate_W}, we can show $\abs{b_{t+1} - b_t} = O(1/t)$ a.s.. Now we invoke \cite[Lemma 8]{Mairal_10} (see Lemma~\ref{lem:conv_nonneg}) to conclude $f_t(\bW_t)-\tilf_t(\bW_t)\convas 0$. 
%the sequence of empirical losses $\{f_t(\bW_t)\}_{t\ge 1}$ converges to the same limit as $\{\tilf_t(\bW_t)\}_{t\ge 1}$ a.s.. 
By Lemma~\ref{lem:regularity} and \cite[Example 19.7]{Vaart_00} (see Lemma~\ref{lem:suff_Donsker}), we can show the class of measurable functions $\{\ell(\cdot,\bW):\bW\in\calC\}$ is $\bbP$-Glivenko-Cantelli \cite{Vaart_00}. By the Glivenko-Cantelli theorem (see Lemma~\ref{lem:GlivenkoCantelli}) we have 
\begin{equation}
\sup_{\bW\in\calC} \abs{f_t(\bW) - f(\bW)} \convas 0. \label{eq:Gliv_Cantelli}
\end{equation}
%that $f_t$ converges uniformly to $f$ a.s.. 
In particular, 
\begin{equation}
f_t(\bW_t) - f(\bW_t)\convas 0. \label{eq:as_conv_diff2}
\end{equation}
Finally, \eqref{eq:as_conv_diff} and \eqref{eq:as_conv_diff2} together imply that  $\{f(\bW_t)\}_{t\ge 1}$ converges to the same almost sure limit as $\{\tilf_t(\bW_t)\}_{t\ge 1}$.
%This essentially implies $\{f(\bW_t)\}_{t\ge 1}$ and $\{\tilf_t(\bW_t)\}_{t\ge 1}$ converge to the same almost sure limit  because  by %See Section~\ref{sec:as_conv} for a detailed proof. 
\end{psketch}




\begin{theorem}[Almost sure convergence of the sequence of dictionaries]\label{thm:local_min}
The stochastic  process $\{\bW_t\}_{t\ge 1}$ converges to the set of stationary points of \eqref{eq:exp_loss} a.s..\footnote{Given a metric space $(\calX,d)$, a sequence $(x_n)$ in $\calX$ is said to converge to a set $\calA\subseteq\calX$ if $\lim_{n\to\infty}\inf_{a\in\calA}d(x_n,a) = 0$.}
\end{theorem}
\begin{psketch}
%By \eqref{eq:as_conv_diff}, it suffices to show that for every realization of $\{\bv_t\}_{t\ge 1}$ such that $\tilf_t(\bW_t) - f_t(\bW_t)\to 0$, each cluster point of $\{\bW_t\}_{t\ge 1}$ is a local minimizer of $f$. 
Fix a realization $\{\bv_t\}_{t\ge 1}$ such that $\tilf_t(\bW_t) - f_t(\bW_t)\to 0$,\footnote{The set of all such realizations have probability one by \eqref{eq:as_conv_diff}.} and generate $\{\bW_t\}_{t\ge 1}$ according to Algorithm~\ref{algo:generic}. Then it suffices to show every subsequential limit of $\{\bW_t\}_{t\ge 1}$ is a stationary point of \eqref{eq:exp_loss}. 
The compactness of $\calC$ enables us to find a convergent subsequence $\{\bW_{t_m}\}_{m\ge 1}$. Furthermore, it is possible to find a further subsequence of $\{t_m\}_{m\ge 1}$, $\{t_k\}_{k\ge 1}$ such that all $\{\bA_{t_k}\}_{k\ge 1}$, $\{\bB_{t_k}\}_{k\ge 1}$ and $\{\tilf_{t_k}(\vecz)\}_{k\ge 1}$ converge.  We focus on $\{\bW_{t_k}\}_{k\ge 1}$ hereafter and denote its limit by $\barbW$. Our goal is to show  for any $\bW\in\calC$, the directional derivative $\lrangle{\nabla f(\barbW)}{\bW - \barbW}\ge 0$. First, we show the sequence of differentiable functions $\{\tilf_{t_k}\}_{k\ge 1}$ converges uniformly to a differentiable function $\barf$. By \eqref{eq:Gliv_Cantelli} we also have that $\{f_{t_k}\}_{k\ge 1}$ converges uniformly to $f$. Denote $g_t \defeq \tilf_t - f_t$, we have $\{g_{t_k}\}_{k\ge 1}$ converges uniformly to $\barg = \barf - f$. By Lemma~\ref{lem:regularity}, $f$ is differentiable so $\barg$ is differentiable on $\calC$. Next, we show for any $\bW\in\calC$, $\lrangle{\nabla \barf(\barbW)}{\bW - \barbW}\ge 0$ by showing $\barbW$ is a global minimizer of $\barf$. Then we show $\nabla \barg(\barbW) = \vecz$ using the first-order Taylor expansion. These results suffice to imply $\lrangle{\nabla f(\barbW)}{\bW - \barbW}\ge 0$ for any $\bW\in\calC$. 
\end{psketch}

\begin{remark}\label{rmk:nonconvexity}
Due to the nonconvexity of the canonical NMF problem, finding global optima of %the canonical NMF problem
 \eqref{eq:canon_NMF} is in general out-of-reach. Indeed, \cite{Vavasis_09} shows \eqref{eq:canon_NMF} is NP-hard. 
 Therefore in the literature \cite{Lin_07b,Haji_16}, convergence to the set of stationary points (see Definition~\ref{def:stat_pt}) of \eqref{eq:canon_NMF}  has been studied instead. In the online setting, it is even harder to  show that the sequence of dictionaries $\{\bW_t\}_{t\ge 1}$ converges to the global optima of \eqref{eq:exp_loss} (in some probabilistic sense). Thus Theorem~\ref{thm:local_min} only states the convergence result with respect to the stationary point of \eqref{eq:exp_loss}, which has been the state-of-the-art in the literature of online matrix factorization \cite{Mairal_10,Shen_14b}. On the other hand, we notice that under certain assumptions on the data matrix $\bV$, e.g., the separability condition proposed in \cite{Donoho_04}, exact NMF algorithms have been proposed in~\cite{Cohen_93,Arora_12,Gillis_12}. The optimization techniques therein are discrete (and combinatorial) in nature, and vastly differ from the continuous optimization techniques typically employed in solving the canonical NMF problem. In addition, some heuristics for obtaining exact NMF using continuous optimization techniques have been proposed in \cite{Vanda_16}. Nevertheless, developing exact NMF algorithms in the online setting could be an interesting future research direction.
\end{remark}

The following two lemmas are used in the proof of Theorem~\ref{thm:as_conv} and \ref{thm:local_min}. In particular, Lemma~\ref{lem:regularity} states that the loss functions $\ell$ and $f$ respectively defined in \eqref{eq:ith_loss} and \eqref{eq:exp_loss} satisfy some regularity conditions.
% w.r.t. a single sample, $\ell$ and the expected loss $f$. 
Lemma~\ref{lem:rate_W} then bounds the variations in the stochastic process $\{\bW_t\}_{t\ge 1}$. 
%provides sufficient conditions to invoke technical theorems (for e.g., Donsker and Glivenko-Cantelli) in the proof. Lemma~\ref{lem:rate_W} bounds the variations between consecutive dictionary matrices $\bW_{t+1}$ and $\bW_t$. %Although it does not suffice to prove the almost sure convergence of $\{\bW_t\}_{t\ge 1}$, 
%When coupling with the Lipschitz continuity of loss functions (for e.g., $f_t$ and $\tilf_t$), it bounds the variations in the corresponding processes (for e.g., $\{f_t(\bW_t)\}_{t\ge 1}$ and $\{\tilf_t(\bW_t)\}_{t\ge 1}$), thereby leading to the convergence results in Theorem~\ref{thm:as_conv}. %Moreover, these  Since they may be of independent interest, we also present them below. 

\begin{lemma}[Regularity properties  of $\ell$ and $f$]\label{lem:regularity}\quad
%We prove regularities of two important loss functions, $\ell(\bv,\bW)$ and $f(\bW)$
The loss functions $\ell$ and $f$ satisfy the following properties\\
1) $\ell(\bv,\bW)$ is differentiable on $\calV\times\calC$ and $\nabla_\bW\ell(\bv,\bW)$ is continuous on $\calV\times \calC$. Moreover, for all $\bv\in\calV$, $\nabla_\bW\ell(\bv,\bW)$ is Lipschitz on $\calC$ with Lipschitz constant independent of $\bv$. \\
2) The expected loss function $f(\bW)$ is differentiable on $\calC$ and $\nabla f(\bW)$ is Lipschitz on $\calC$.
\end{lemma}
\begin{psketch}
The regularity properties of $\ell$ on $\calV\times\calC$ originate  from the regularity properties  of $\tilell$ on $\calV\times\calC$. Since Assumption~\ref{assum:sc_hr} ensures the minimizer for \eqref{eq:new_ithloss}, $(\bh^*(\bv,\bW),\br^*(\bv,\bW))$ is unique for any $(\bv,\bW)\in\calV\times\calC$, we can invoke Danskin's theorem (see Lemma~\ref{lem:min_diff}) to guarantee the differentiability of $\ell$. Moreover, we invoke the maximum theorem (see Lemma~\ref{lem:min_cont}) to ensure the continuity of  $(\bh^*(\bv,\bW),\br^*(\bv,\bW))$ on $\calV\times\calC$. These results together imply that $\ell$ satisfies the desired regularity properties. The regularity properties of $f$ on $\calC$ hinges upon the regularity properties  of $\ell$. Indeed, by  Leibniz integral rule (see Lemma~\ref{lem:Leib_int}), we can show $f$ is differentiable and 
\begin{equation}
\nabla f(\bW) = \bbE_\bv[\nabla_\bW \ell(\bv,\bW)]. \label{eq:grad_f}
\end{equation} 
Hence the Lipschitz continuity of $\nabla f$ follows naturally from the Lipschitz continuity of $\bW\mapsto\nabla_\bW\ell(\bv,\bW)$. %See Section~\ref{sec:regularity} for a detailed proof. 
\end{psketch}


\begin{lemma} \label{lem:rate_W}
In Algorithm~\ref{algo:generic}, $\norm{\bW_{t+1}-\bW_t}_F = O\left(1/t\right)$ a.s..
\end{lemma}
\begin{psketch}
The upper bound for $\norm{\bW_{t+1}-\bW_t}_F$ results from the strong convexity and Lipschitz continuity of $\tilf_t$. Specifically, they together imply an order difference between the upper and lower bounds of $\tilf_t(\bW_{t+1}) - \tilf_t(\bW_t)$, i.e., 
\begin{align}
\frac{m_2}{2}\norm{\bW_{t+1}-\bW_t}_F^2& \le \tilf_t(\bW_{t+1}) - \tilf_t(\bW_t) \\
& \le c_t \norm{\bW_{t+1}-\bW_t}_F, %\label{eq:upp_lower_tilf_t}
\end{align}
where $c_t$ is only dependent on $t$. Some calculations reveal that $c_t = O(1/t)$ a.s.. %See Section~\ref{sec:rate_W} for a detailed proof. 
\end{psketch}


%The following lemma is used in the proof of Lemma~\ref{lem:regularity}. Since it may be of independent interest, we also present it below. 
%\begin{lemma} \label{lem:Lips_b}
%Let $\bz$, $\bz' \in\calZ\subseteq \bbR^m$ and $\bA$, $\bA'\in\calA\subseteq \bbR^{m\times n}$, where both $\calZ$ and $\calA$ are compact sets. Let $\calB$ be a compact set in $\bbR^n$, and define $g:\calB\to\bbR$ as $g(\bb) = \norm{\bz-\bA\bb}_2^2 - \norm{\bz'-\bA'\bb}^2_2$. Then $g$ is Lipschitz on $\calB$ with constant $c_1\norm{\bz-\bz'}_2 + c_2\norm{\bA-\bA'}_2$ where $c_1$ and $c_2$ are two positive universal constants. In particular, $\tilg(\bb) = \norm{\bz-\bA\bb}_2^2$ is Lipschitz on $\calB$ with a universal Lipschitz constant $c>0$. %By symmetry, we also have similar results for $h(\bA) = \norm{\bz-\bA\bb}_2^2 - \norm{\bz'-\bA\bb'}^2_2$, where $\bb,\bb'\in\calB$. 
%\end{lemma}
%\begin{psketch}
%This lemma can be proved by straightforward calculations. %See Section~\ref{sec:Lips_b} for a detailed proof.  
%\end{psketch}

\subsection{Discussions}\label{sec:conv_dis}
%\subsubsection{Innovations and novelty}
We highlight the distinctions between our convergence analyses and those in the previous works. %In this section, we have proved the convergence results regarding $\{f(\bW_t)\}_{t\ge 1}$ and $\{\bW_t\}_{t\ge 1}$ (shown in Theorem~\ref{thm:as_conv} and \ref{thm:local_min} respectively). %The first result (almost sure convergence of $f(\bW_t)$), Our first result concerns the almost sure convergence of $\{f(\bW_t)\}_{t\ge 1}$. 
Our first main result (Theorem~\ref{thm:as_conv}) is somewhat analogous to the results in  \cite{Mairal_10,Feng_13,Shen_14b,Guan_12}. % although the exact definition of $\{f(\bW_t)\}_{t\ge 1}$ may differ from one work to another.  
However, the stochastic optimization of \eqref{eq:exp_loss} in \cite{Guan_12} is based on   robust stochastic approximations \cite{Nemi_09}, a very different approach from ours (see Section~\ref{sec:algo}). This leads to significant differences in the analyses. For example, at time $t$, their dictionary update step does not necessarily minimize $\tilf_t(\bW)$, a fact that we heavily leverage. The rest of the works fall under a similar framework as ours. However, in \cite{Mairal_10}, a general sparse coding problem is considered. Thus, they assume a sufficient condition ensuring a unique solution in the Lasso-like sparse coding step holds. This assumption adds certain complications in proving $\nabla f$ is Lipschitz. Because our problem setting is different, we avoid these issues. Thus we are able to provide a more succinct proof, despite having some additional features, e.g., the presence of the outlier vector $\br$. The most closely related works are~\cite{Feng_13} and \cite{Shen_14b}, both of which consider the online robust PCA problems but with different  loss functions. %Although certain similarities are shared between our proof and theirs, 
However, the nonnegativity constraints on $\bW$ and $\bh$ and box constraints on $\br$ distinguish our proof from theirs. 
%(How to say this properly?) Previous works \cite{Mairal_10,Feng_13,Shen_14b} also proved $\{\bW_t\}_{t\ge 1}$ converges a.s.\ to a stationary point of $f$ via similar approaches. However, in our case, we are not able to show $\{\bW_t\}_{t\ge 1}$ converges a.s.\ or even $\{\nabla f(\bW_t)\}_{t\ge 1}$ converges a.s. using an analogous approach. As clarified by an author of \cite{Mairal_10}, only a subsequence of $\{\bW_t\}_{t\ge 1}$ converges. 
For our second main result (Theorem~\ref{thm:local_min}), we note that in   previous works %on other matrix factorization problems 
\cite{Feng_13,Shen_14b}, seemingly stronger results have been shown. %That is, the sequence $\{\bW_t\}_{t\ge 1}$ converges to a stationary  point (or the global minimizer in ) of $f$. 
Specifically, these works manage to show $\{\bW_t\}_{t\ge 1}$ converges, either to {\em one} stationary point \cite{Shen_14b} or to the global minimizer \cite{Feng_13} of $f$.
However, due to different problem settings, their proof techniques cannot be easily adapted to our problem setting. Inspired by \cite{Mairal_13}, we instead manage to characterize the subsequential limits of $\{\bW_t\}_{t\ge 1}$ generated by almost sure realizations of $\{\bv_t\}_{t\ge 1}$. This result is almost as good as the results in the abovementioned previous works since it implies that there exists a stationary point of $f$ such that its neighborhood contains infinitely many points of the sequence almost surely. 

%\subsubsection{Sufficient Conditions for Assumption~\ref{assum:sc_W}} \label{sec:rmv_scW}
%Let us now choose a different upper-bound function for $f_t$ on $\calC$, $\tilf'_t$ as
%\begin{align*}
%&\tilf_t'(\bW) \defeq \frac{1}{t}\sum_{i=1}^t\frac{1}{2}\norm{\bv_i-\bW\bh_i-\br_i}_2^2+\lambda\norm{\br_i}_1 \\
%&\hspace{5cm}+ \frac{m_2}{2}\norm{\bW-\bW_{i-1}}_F^2.
%\end{align*}
%Clearly $\tilf'_t$ is $m_2$-strongly convex on $\calC$.
%Compared to $\tilf_t$, we observe that $\tilf_t'$ has an additional proximal term $m_2/(2t)\sum_{i=1}^t\norm{\bW-\bW_{i-1}}_F^2$. %, which makes it $m_2$-strongly convex on $\calC$. 
%We remark that adding this proximal term has different effects as compared with adding the simple Tikhonov term $m_2/2\norm{\bW}_F^2$ (as discussed in Remark~\ref{rmk:assump}). Specifically, adding the proximal term enables us to use essentially the same techniques to prove the almost sure convergence of Algorithm~\ref{algo:generic}. %In particular, Lemma~\ref{lem:rate_W} will remain unchanged  %
%(Details are omitted in the interest of space.)
%However, the proof techniques in this paper may not be directly applicable for the Tikhonov case. 
%
%In terms of the algorithm, the dictionary update step \eqref{eq:dict_upd} in Algorithm~\ref{algo:generic}, which now amounts to minimizing $\tilf_t'$ over $\calC$, can be written compactly as
%\begin{equation*}
%\bW_t := \frac{1}{2}\tr\left(\bW^T\bW(\bA_t+m_2\bI)\right) - \tr\left(\bW^T(\bB_t+m_2\bC_t)\right),
%\end{equation*}
%where $\bC_t\defeq 1/t\sum_{i=1}^t\bW_{i-1}$ is another sufficiently statistic. %Since our convergence analyses only capitalize on the upper-bound property of $\tilf_t$,  replacing $\tilf_t$ with $\tilf'_t$ will not alter the analyses in most aspects. 
%By defining $\bA'_t \defeq \bA_t+m_2\bI$ and $\bB'_t \defeq \bB_t+m_2\bC_t$, the modified algorithm consumes the same storage space as the original one. 
%Therefore by replacing $\tilf_t$ with $\tilf'_t$, we manage to remove Assumption~\ref{assum:sc_W}. 

% for sufficiently large $t$, there always exists a local minimizer of $f$ to which $\bW_t$ is arbitrarily ``close'' 
%%to the set of of local minimizers of $f$
%almost surely. 
\section{Extensions} \label{sec:extension}
In Section~\ref{sec:prob}, we considered a basic formulation of the online NMF problem with outliers. Here, %based on the literature in (online) NMF, 
we extend this formulation to a wider class of problem settings, with different constraints and regularizers on $\bW$, $\bh$ and $\br$. We also discuss how to adapt our algorithms and analyses to these settings. 
%Moreover, we also discuss some other possible methodologies to tackle the stochastic program \eqref{eq:exp_loss}.

\subsection{Different Constraints on $\bW$ and $\br$}
If the constraint set $\calC$ changes, in terms of both solvers in Section~\ref{sec:algo},   %(based on PGD and ADMM), 
only \eqref{eq:proj_W} and \eqref{eq:ADMMupd_Q}, the updates involving projections onto $\calC$, need to be changed. In terms of the analyses, using a similar argument as in Section~\ref{sec:prelim}, we can show the (unique) optimal solution $\bW^*(\bA_t,\bB_t)$ in \eqref{eq:min_W_tr} is bounded\footnote{From \eqref{eq:min_W_tr}, we observe that $\bW_t$ is a function of only $\bA_t$ and $\bB_t$ if Assumption~\ref{assum:sc_W} holds.} regardless of boundedness of $\calC$. Thus, the optimization problem \eqref{eq:min_W_tr} can always be restricted to a compact set. The rest of the analyses proceeds as usual. Therefore, in the following, we only discuss the (convex) variants of $\calC$ and the associated (Euclidean) projection methods. 
%no matter if $\calC$ is bounded. 
%Other constraint sets on $\bW$ include

1) The nonnegative orthant  $\calC_1 \defeq \bbR_+^{F\times K}$. This constraint is the most basic yet the most widely used constraint in NMF, although it does not well control the scale of $\bW$. The projection onto the nonnegative orthant involves simple entrywise thresholding. 

2) The probability simplex $\calC_2 \defeq \{\bW\in\bbR_+^{F\times K}\,|\,\norm{\bW_{:i}}_1= 1, \forall\,i\in[K]\}$ \cite{Guan_12}. Different from $\calC$, $\calC_2$ also prevents some columns of $\bW$ from having arbitrarily small norms. Efficient projection algorithms onto the probability simplex have been well studied. See \cite{Duchi_08,WangSimp_13} for details. 

3) The nonnegative elastic-net ball \cite{Witten_09,Zou_05,Sind_12}:  $\calC_3 \defeq \{\bW\in\bbR_+^{F\times K}|\gamma_1\norm{\bW_{:i}}_1 + \gamma_2/2\norm{\bW_{:i}}_2^2 \le 1, \forall\,i\in[K]\}$, where both $\gamma_1$ and $\gamma_2$ are {\em nonnegative}. $\calC_3$ is a general constraint set since it subsumes both $\calC$ (the nonnegative $\ell_2$ ball) and the nonnegative $\ell_1$ ball. Compared to $\calC$, this constraint encourages sparsity on the basis matrix $\bW$, thus leading to better interpretability on the basis vectors. Efficient algorithms for projection onto the elastic-net ball have been proposed in \cite{Duchi_09,Mairal_10}.


For the outlier vector $\br$, the nonnegativity constraint can be added to $\calR$ to model (bounded) nonnegative outliers, so the new constraint $\calR'\defeq\{\br\in\bbR_+^{F}\,|\,\norm{\br}_\infty\le M\}$. In such case $\norm{\br}_1 = \bone^T\br$ so \eqref{eq:min_r} amounts to a quadratic minimization program with box constraints. %The algorithms for solving such a problem are straightforward and the analyses remains unchanged.
Both the algorithms and the analyses can be easily adapted to such a simpler case. 

\subsection{Regularizers for $\bW$, $\bh$ and $\br$} 
We use $\lambda_i\ge 0$ to denote the penalty parameters. 
For $\bW$, the elastic-net regularization can be employed, i.e., $\lambda_1\norm{\bW}_{1,1} + \lambda_2/2\norm{\bW}_F^2$. %where $\lambda_1$ and $\lambda_2$ are nonnegative penalty parameters. 
This includes   $\ell_1$ and $\ell_2$ regularizers on $\bW$ as special cases. As pointed out in \cite{WuShen_14,Kim_12} and \cite{Witten_09}, $\ell_1$ and $\ell_2$ regularizers promote sparsity and smoothness on $\bW$ respectively. Because  $\bW\ge 0$, \eqref{eq:min_W_tr} with the elastic-net regularization is still quadratic. In terms of the analyses, Assumption~\ref{assum:sc_W} can be removed and the rest remains the same. 

For $\bh$, several regularizers can be used:

1) The %$\ell_1$ (Lasso) regularizer 
Lasso regularizer $\lambda_3 \norm{\bh}_1$. %This is a common regularizer in the literature for inducing sparsity on $\bh$. 
It induces sparsity on $\bh$, hence the  optimization problem \eqref{eq:min_h} with such regularizer is termed {\em nonnegative sparse coding}\cite{Sind_12,Wang_13}. % Thus  becomes a {\em nonnegative Lasso} problem, which 
%and is quadratic in nature. 

2) The Tikhonov regularizer: $\lambda_4/2 \norm{\bh}_2^2$. This regularizer induces smoothness and avoids over-fitting on $\bh$. Both the $\ell_1$ and $\ell_2$ regularizers preserve the quadratic nature of \eqref{eq:min_h}.
%When combined with the $\ell_1$ regularizer, it becomes the elastic-net regularizer, which has been shown to promote group sparsity on $\bh$ \cite{Zou_05}. 

3) The group Lasso regularizer $\lambda_5 \sum_{\alpha} \sqrt{\xi_\alpha}\norm{\bh_\alpha}_2$, where $\bh_\alpha$'s are (non-overlapping) subvectors of $\bh$ with lengths $\xi_\alpha$. This regularizer induces sparsity among groups of entries in $\bh$. Efficient algorithms to solve \eqref{eq:min_h} under this regularization have been proposed in \cite{Liu_11,Kim_12}. 

4) The %$\ell_{2,1} + \ell_1$ (Sparse Group Lasso) 
 sparse group Lasso regularizer $\lambda_6 (\nu\sum_{\alpha} \sqrt{\xi_\alpha}\norm{\bh_\alpha}_2$ $ + (1-\nu)\norm{\bh}_1)$, where $\nu\in[0,1]$. Compared with the group lasso regularizer, this one also encourages sparsity within each group. Efficient algorithms for solving \eqref{eq:min_h} with this regularization have been  discussed in \cite{Friedman_10,Simon_13}.
%As pointed out in \cite{Zou_05}, it can enhance the performance of the Lasso under the high-dimensional setting or highly correlated predictors. 

Since \eqref{eq:min_h} with each regularizer above is still a convex program, the analyses remain unchanged. 
Similar regularizers on $\bh$ can be applied to $\br$, so for simplicity we omit the discussions here. Compared to $\bh$, the only differences are that $\br$ may be negative and bounded. However, standard algorithms in such case are available in the literature \cite{Yuan_06,Simon_13,Friedman_10}. 

\begin{remark} \label{rmk:cons}
Some remarks are in order. First, for certain pairs of constraints and regularizers above (e.g., the elastic-net ball constraint of $\bW$ and the elastic-net regularization on $\bW$), the optimization problems are equivalent (admit the same optimal solutions) for specific value pairs of $(\gamma_i,\lambda_i)$. However, the algorithms and analyses for these equivalent problems can be much different. Second, all the alternative constraint sets and regularizers discussed above are convex, since our analyses heavily leverage the (strong) convexity of $\tilell$ and $\tilf$ (see Section~\ref{sec:prelim}). It would be a meaningful future research direction to consider   nonconvex constraints or regularizers.
%for e.g., the (nonnegative) unit $\ell_2$ sphere or the $\ell_0$ regularizer. 
\end{remark}

%\subsection{Other methodologies to optimize \eqref{eq:exp_loss} }
%From \eqref{eq:grad_f}, we observe that a natural strategy for optimizing \eqref{eq:exp_loss} is the {\em stochastic gradient descent} (SGD) method \cite{Bottou_10,Roux_12}. Specifically, at time $t$, we update the basis matrix $\bW_t$   
%\begin{equation*}
%\bW_t = \calP_\calC\left(\bW_t - \zeta_t{\nabla_\bW \ell(\bv_t,\bW_{t-1})}\right),
%\end{equation*}
%where $\zeta_t>0$ is the step size and 
%\begin{align*}
%&\nabla_\bW\ell(\bv_t,\bW_{t-1}) = \bW_{t-1}\bh^*(\bv_t,\bW_{t-1})\nn\\
%&\hspace{4.5cm} + \br^*(\bv_t,\bW_{t-1})-\bv_t.
%\end{align*}
%Compared with our current approach, this method need not store the sufficient statistics $\bA_t$ and $\bB_t$, thus can be applied to the scenario where the fit-to-data term between $\bv$ and $\bW\bh+\br$, $d(\bv\Vert \bW\bh+\br)$ is not the Frobenius loss, for e.g., the $\beta$-divergence \cite{Fev_09,Fev_11}. However, the step size $\zeta_t$ needs to be judiciously chosen \cite{Rob_51,Nemi_09,Mairal_10} in order for the SGD method to have fast convergence. Other variants such as the {\em averaging} stochastic gradient descent \cite{Polyak_92} and the {\em second-order} stochastic gradient descent \cite{Bottou_10} can also be applied to accelerate the convergence of the SGD algorithm. 
%Moreover, recently some other stochastic optimization frameworks have been proposed to solve \eqref{eq:exp_loss}, for e.g., the stochastic majorization-minimization algorithm \cite{Mairal_13}. However, this approach requires building a suitable convex smooth surrogate function for \eqref{eq:ith_loss}, thus is more complicated to implement than the SGD approach. 


\section{Experiments} \label{sec:numericals}

%\subsection{Datasets}
We  present results of numerical experiments to demonstrate the computational efficiency and efficacy of our online algorithms (OPGD and OADMM). We first state the experimental setup. Next we introduce some heuristics used in our experiments and the choices of parameters. We show the efficiency of our algorithms by examining the convergence speeds of the objective functions. The efficacy of our algorithms is shown via the (parts-based) basis representations and three meaningful real-world  applications---image denoising, shadow removal and foreground-background separation.  
All the experiments were run in 64-bit $\mbox{Matlab}^\circledR$ (R2015b) on a machine with $\mbox{Intel}^\circledR$ $\mbox{Core}$ i7-4790 3.6 GHz CPU and 8 GB RAM. 
%The codes for all the experiments are implemented in $\mbox{Matlab}^\circledR$ (2015b) and run on a 3.4 GHz Intel i7-GPU with 8 GB RAM. 

\subsection{Experimental Setup}
The datasets used in the experiments include one synthetic dataset, two face datasets (including the {\tt CBCL} dataset \cite{Lee_00} and the {\tt YaleB} dataset \cite{YaleB_01}) and two video sequences in the {\tt i2r} dataset\cite{i2r_04}. %both of which are commonly used in benchmarking NMF and related algorithms. Different tasks were performed on different datasets. 
For each task, we compared the performances of our online algorithms against those of four other matrix factorization algorithms, including BPGD, BADMM, online robust PCA (ORPCA) \cite{Feng_13} and online NMF (ONMF) \cite{Guan_12}. Such comparisons allow us to show multiple advantages of the proposed algorithms. Specifically, compared with their batch counterparts (BPGD and BADMM), our online algorithms have much faster convergence speeds in terms of the objective values.\footnote{\label{fn:online_batch} The problem formulations in the existing batch robust NMF algorithms \cite{Gao_15, Huang_14, Yang_13, Du_12, Ding_12, Ding_11, Cichoc_11, Fev_14,Shen_14,Zhang_11} are different from ours here. Specifically, most of the algorithms do not use the $\ell_{1,1}$ regularization to remove the effect of outliers. Moreover, the constraints on $\bW$ and $\bR$ are simpler in their formulations. All these differences contribute to different results in the experiments. %Thus to solely demonstrate the computational efficiency of the online framework, 
Thus for the purpose of fair comparisons, we chose to compare our online algorithms with their batch counterparts.}
 Moreover, compared with ORPCA, %our online algorithms 
we are able to learn parts-based basis representations. Finally, compared with ONMF, we can recover the underlying low-dimensional data structure with minimal effects of outliers. 
%These results include convergence speed of objective values, 
%We present our experiment results in the order of datasets, which 
%As described in Section~\ref{sec:algo}, we have effectively proposed two algorithms, namely ONMFO based on ADMM and ONMFO based on PGD. For ease of reference, we term these two algorithms as OADMM and OPGD respectively. Moreover, we call their batch counterparts as BADMM and BPGD respectively as discussed in Remark~\ref{rmk:algo}. %(see Section~\ref{sec:BRNMF_algo} in the supplemental material for detailed descriptions). 
%As a class of matrix factorization algorithms, our algorithms have several advantages, including low computational (and storage) complexity, parts-based basis representations and outlier-robustness. %being robust to outliers. 
%To demonstrate each of the advantages, we will compare our online algorithms (OADMM and OPGD) with four algorithms from three classes. The first two algorithms are BADMM and BPGD, since both of them are batch NMF algorithms designed to handle outliers. We will show our online algorithms have much faster convergence speed compared to their batch counterparts

\subsection{Strategies in Practical Implementations} \label{sec:heuristics}
%In the literature of online matrix factorization \cite{Mairal_10,Bucak_08}, several heuristic methods have been employed to pre-process the datasets and speed up the convergence of the online algorithms. In our experiments, we have used some of these heuristic methods. 
%In this section we first describe our methods to choose the initial dictionary $\bW_0$ and the stopping criteria for solving the sub-optimization problems \eqref{eq:learn_hr} and \eqref{eq:dict_upd} in Algorithm~\ref{algo:generic}. Then we describe several heuristics employed in the implementations of our online algorithms. %These heuristics are common in the 
\subsubsection{Initializations} \label{sec:init}
For both online algorithms (OPGD and OADMM), the entries of initial dictionary $\bW_0$ were randomly generated independently and identically from the standard uniform distribution $\scU[0,1]$. This distribution can certainly be of other forms, e.g.,  the half-normal distribution\footnote{$\scHN\left(\sigma^2\right)$ denotes the half-normal distribution with scale parameter $\sigma^2$, i.e., $\scHN(y ; \sigma^2)=\frac{\sqrt{2}}{\sigma\sqrt{\pi}}\exp(-\frac{y^2}{2\sigma^2})$ for $y\ge 0$. If $Y\sim\calN(0,\sigma^2)$, $|Y|$ is half-normal with scale parameter $\sigma^2$.} $\scHN(0,1)$. However, we observed in all the experiments that different initialization schemes of $\bW_0$ led to similar results. At each time instant, similar initialization methods on $\bh$ and $\br$ were also used in solving \eqref{eq:learn_hr}. 
While for solving \eqref{eq:dict_upd}, we use $\bW_{t-1}$ as the initialization to exploit the similarities between dictionaries in adjacent iterations. This approach led to improved computational efficiency in practice. 
For the batch algorithms (BPGD and BADMM), the initial dictionary and coefficient matrix were similarly initialized as $\bW_0$.

\subsubsection{Stopping Criteria}\label{sec:stop_crit} 
For the optimization problem \eqref{eq:learn_hr},  we employed the same stopping criterion for both algorithms OPGD and OADMM. Specifically, at time $t$, we stopped \eqref{eq:learn_hr} at iteration $k$ if 
\begin{equation}
\frac{\abs{\tilell(\bv_t,\bW_{t-1},\bh^k,\br^k) - \tilell(\bv_t,\bW_{t-1},\bh^{k-1},\br^{k-1})}}{\tilell(\bv_t,\bW_{t-1},\bh^{k-1},\br^{k-1})}< \epsilon_{\rmth}\nn
\end{equation}
or $k>\varkappa_{\max}$, where $\epsilon_{\rmth}>0$ denotes the threshold for the relative decrease of objective values and $\varkappa_{\max}\in\bbN$  denotes the maximum number of iterations. We set $\epsilon_{\rmth}=1\times 10^{-3}$ and $\varkappa_{\max}=50$ for all the experiments. Similar stopping criterion applies to both  OPGD and OADMM in solving \eqref{eq:dict_upd}, except in this case, we set $\epsilon_{\rmth}=1\times 10^{-4}$ and $\varkappa_{\max}=200$ for better precision in updating the dictionary. 

\subsubsection{Heuristics}
We now describe some heuristics employed in the implementations of our online algorithms. We remark that all the heuristics used in this work are common in the literature of online matrix factorization \cite{Mairal_10,Bucak_08}.
First, since the sample size of the {\em benchmarking} datasets in real world may not be large enough, %are typically of small or medium scale (i.e., the number of data samples in a dataset are of order $10^3$ or $10^4$), 
we replicate it $p$ times to aggregate its size. %This is because  especially the image datasets, .\footnote{There are some large text datasets, for e.g., the UCI Bag of Words dataset \cite{UCIBoW_08}. However, in general it is very hard to learn parts-based basis representations on these datasets.} 
%Thus, this replication serves as one simple way to create  or simulate large-scale datasets.  
Each (aggregated) dataset is represented as an $F\times N$ matrix, where $F$ equals the ambient dimension of the data\footnote{Each image (or video frame) is vectorized and stacked as a column of the data matrix, so $F$ equals the number of pixels in the image (or video frame).}  and $N$ equals $p$ times the size of the original dataset.
Next, for the aggregated image dataset, we randomly shuffle the images in it to simulate an \iid data stream. We also normalize each data sample so that it has unit maximum pixel value. %At time $t$, for both OPGD and OADMM algorithms, 
The last heuristic is called the mini-batch extension, namely at a time instant, we process $\tau$ images  altogether ($\tau\ge 1$). This amounts to sampling $\tau$ \iid samples from $\bbP$ at each time. 
%Moreover, since we only have guarantee for asymptotic almost sure convergence, the online algorithms may not converge to a pre-specified accuracy 

%There are in total  parameters whose values need to be set in our online algorithms. 
\subsection{Parameter Settings} 
We   discuss how to set the values of the important parameters in our online algorithms. 
These parameters include the regularization parameter $\lambda$ in \eqref{eq:ith_loss}, the constraint parameter $M$ in the definition of $\calR$, the mini-batch size $\tau$, the latent data dimensionality $K$,  the penalty parameters $\{\rho_i\}_{i\in[3]}$ %$\rho_1$, $\rho_2$ and $\rho_3$ 
in OADMM, and the step sizes $\barkappa$ and $\{\tkappa_t\}_{t\ge 1}$ in OPGD. First, we set $\lambda = 1/\sqrt{F}$, following the convention in the literature \cite{Mairal_10,Feng_13,Shen_14b}. Since each image has unit maximum pixel value, it is straightforward to set $M=1$. %\footnote{As shown below, we also constrain the maximum entry of the (nonnegative) synthetic data matrix to be one so the same value of $M$ also works for the synthetic data.} 
For the mini-batch size $\tau$, we propose a rule-of-thumb, that is to choose $\tau \le 4\times 10^{-4}N$, e.g., $\tau = 5\times 10^{-5}N$. The choice of this parameter involves a trade-off between the stability of the basis matrix  and the convergence speed of the online algorithms. In general this parameter is data-dependent so there is no principled way of choosing it in the literature~\cite{Mairal_10,Guan_12}. %\footnote{This parameter may depend on the sample complexity of the online matrix factorization algorithms. This may be a meaningful research topic in the future.} 
Therefore, our approach presents a heuristic way to partially resolve this issue. Regarding the latent data dimensionality $K$, there are some works \cite{Tan_13a,Bishop_98}  that   describe how to choose this parameter from a Bayesian perspective. However, they involve complex Bayesian modeling and intensive computations. Thus for efficiency we set $K=49$.  Finally we discuss the choice of the parameters specific to each online algorithm. We first reduce the number of parameters in both algorithms by setting %For the three penalty parameters in OADMM, for simplicity we set 
$\rho_i =\rho$, for any $i\in[3]$ in OADMM\footnote{See the literature on ADMM (e.g., \cite{Boyd_11}) for  more sophisticated ways of choosing this parameter.}  and $\barkappa = \tkappa_t= \kappa$, for any $t\ge 1$ in OPGD. Then we set $\rho = 1$ and $\kappa = 0.7$. 
The above parameter setting will be referred as {\em the canonical parameter setting} in the sequel. 
Later we will show the convergence speeds of both our online algorithms are insensitive to the parameters that are set to fixed values (including $\tau$, $K$, $\rho$ and $\kappa$) %$\{\rho_i\}_{i\in[3]}$, $\barkappa$ and $\{\tkappa_t\}_{t\ge 1}$) 
within wide ranges on both the synthetic and {\tt CBCL} datasets. 
\subsection{Convergence Speeds}\label{sec:conv_spd}
%We first compare 
The convergence speeds of the objective values  of OPGD and OADMM  are compared to the other four benchmarking algorithms (BPGD, BADMM, ORPCA and ONMF) on both the synthetic and {\tt CBCL} datasets. For better illustration, the comparison is divided into two parts. The first   involves 
comparison between our online algorithms and their batch counterparts. The second   involves comparison between our online algorithms to other online algorithms. 
For the first part, we use the surrogate loss function $\tilf_t$ as a measure of convergence  because i) $\tilf_t(\bW_t)$ has the same a.s.\ limit as $f(\bW_t)$ (as $t\to\infty$) and ii)   with storage of the past statistics $\{\bv_i,\bh_i,\br_i\}_{i\in[t]}$, $\tilf_t$ is easier to compute than the expected loss $f$ or the empirical loss $f_t$. For the second part, since different online algorithms have different objective functions, we propose a heuristic and unified measure of convergence $\hatf_t:\calC\to\bbR_+$, defined as 
\begin{equation}
\hatf_t(\bW) \defeq \frac{1}{t} \sum_{i=1}^t \frac{1}{2}\norm{\bv^o_{i} - \bW\bh_i}_2^2,
\end{equation}
where $\bv^o_i$ denotes the clean data sample (without outliers and observation noise) at time $i$. 
%Given a basis matrix $\bW$ such that each $\bv^o_i$ lies in its column space, $\hatf_t(\bW)$ can be interpreted as the averaged regret of reconstructing the clean data up to time $t$ using $\{\bh_i\}_{i\in[t]}$. 
Loosely speaking, $\hatf_t(\bW_t)$ can be interpreted as the averaged regret of reconstructing the clean data up to time $t$.
In the following experiments, unless compared with other online algorithms, we always use $\tilf_t$ as the objective function for our online algorithms. 
%the objective functions used in the experiments. For the batch algorithms, the objective functions are well-defined. However, for the online algorithms, we cannot explicitly calculate the expected loss $f$ since we have no access to the data distribution $\bbP$. Indeed, all the online algorithms can only produce a sequence of basis matrices $\{\bW_t\}_{t\ge 1}$, thereby making the objective functions difficult to calculate.
%Here we use  since $\tilf_t(\bW_t)$ has the same almost sure limit as $f(\bW_t)$ (as $t\to\infty$) and it is easy to compute (with storage of past statistics $\{\bv_i,\bh_i,\br_i\}_{i\in[t]}$).  
%define a pseudo-objective function $\hatf_t:\calC\to\bbR$ as follows
%\begin{equation}
%\hatf_t(\bW) = \frac{1}{t}\sum_{i=1}^t \frac{1}{2}\no 
%\end{equation} 
\subsubsection{Generation of Synthetic Dataset}
The procedure to generate the synthetic dataset is described as follows. First, we generated $\bW^o\in \bbR_+^{F\times K^o}$ such that $(w^o)_{ij}\iidsim\scHN(1/\sqrt{K^o})$, for any $(i,j)\in[F]\times[K^o]$, where $K^o$ denotes the ground-truth latent dimension. 
%, where $\calH\calN$ denotes the half-normal distribution. 
Similarly, we generated $\bH^o\in\bbR_+^{K^o\times N}$ such that $(h^o)_{ij}\iidsim\scHN(1/\sqrt{K^o})$, for any $(i,j)\in[K^o]\times[N]$. The (clean) data matrix $\bV^o = \calP_{\tilcalV}\left(\bW^o\bH^o\right)$, where  $\tilcalV = [0,1]^{F\times N}$. Note that the normalization (projection) $\calP_{\tilcalV}$ preserves the boundedness      of the data. %We also normalize $\bV_0$ such that $\norm{\bV_0}_{\max}=1$. 
%Next, we describe the procedure of generating the outlier matrix $\bR^o$. 
Then we generated the outlier vectors $\{\br_i\}_{i\in[N]}$. 
First, we uniformly randomly  selected a subset of $[N]$ %the columns of $\bR_0$ 
with size $\floor{\nu N}$ and denoted it as $\calI$, where $\nu\in(0,1)$ denotes the fraction of  columns in $\bV^o$ to be contaminated by the outliers. %Denote the index set of the contaminated columns as $\calI\subseteq[N]$. 
For each $i\in\calI$, we generated the outlier vector $\br_i\in[-1,1]^F$ by first uniformly selecting a support set with cardinality $\floor{\tnu F}$, where $\tnu\in(0,1)$ denotes the outlier density in each data sample. Then the nonzero entries in $\br_i$ were \iid generated from the uniform distribution $\calU[-1,1]$. For each $i\not\in\calI$, we set $\br_i=\vecz$. Define the outlier matrix $\bR^o$ such that $(\bR^o)_{:i} = \br_i$, for any $i\in[N]$. Then the (contaminated) data matrix $\bV = \calP_{\tilcalV} (\bV^o + \bR^o+\bN)$, where $\bN\in\bbR^{F\times N}$ denotes the observation noise matrix with \iid standard normal entries. 

%\subsubsection{Generation of {\tt CBCL} Dataset with Outliers}\label{sec:CBCL_gen}
%For the {\tt CBCL} face dataset, we first replicated it with a factor $p = 50$ (so that the total number of data samples is of the order $10^5$) and randomly permuted the aggregated dataset as described in  Section~\ref{sec:heuristics}. We then contaminated it with outliers in the same manner as for the synthetic dataset. However, we avoided adding observation noise %in the final step 
%since it had been already introduced by the image acquisition process. 

\subsubsection{Comparison to Other Online and Batch NMF Algorithms}
To make a fair comparison, all the algorithms were initialized with the same $\bW_0$ for each parameter setting. Moreover, all   online algorithms had the same mini-batch size $\tau$. The {\em synthetic} data matrix $\bV$ was generated using the parameters values $F = 400$, $K^o = 49$, $N = 1\times10^5$, $\nu = 0.7$ and $\tnu = 0.1$.

The convergence speeds of the aforementioned algorithms under the canonical parameter setting\footnote{For BPGD and BADMM, the step sizes and penalty parameters had the same values as those of the online algorithms. For ORPCA and ONMF, we kept the parameter settings in the original works.} are shown in Figure~\ref{fig:canon}. From Figure~\ref{fig:canon}(a), we observe our online algorithms converge much faster than their batch counterparts. From Figure~\ref{fig:canon}(b), we observe the ORPCA algorithm converges slightly faster than our online algorithms (the time difference is less than 1s). This is because ORPCA's formulation does not incorporate nonnegativity constraints (and   magnitude constraints for the outliers) so the algorithm has fewer projection steps, leading to a lower computational complexity. However, we will show in Section~\ref{sec:basis_rep} that it fails to learn the parts-based   representations due to the lack of nonnegativity constraints. Moreover, we also observe  that the ONMF algorithm fails to converge because it is unable to handle the outliers. Thus the constant perturbation from the outliers in the data samples on the learned basis matrix keeps the algorithm from converging. Furthermore, from both subfigures, we observe the objective function of OADMM has a smoother decrease that that of OPGD. This is OADMM solves the optimization problems \eqref{eq:min_hr} and \eqref{eq:min_W_tr} in the dual space so it avoids the sharp transitions from  plateaus to valleys in the primal space. Apart from this difference, {\em the convergence speeds of OADMM and OPGD are almost the same}. 
\subsubsection{Insensitivity to Parameters}
We  now examine the influences of various parameters, including $\tau$, $K$, $\rho$ and $\kappa$, on the convergence speeds of our online algorithms on the synthetic dataset. To do so, we vary one parameter at a time while keeping the other parameters fixed as in the canonical setting. We first vary $\tau$ from 5 to 40 in a log-scale, since our proposed heuristic rule indicates $\tau\le 40$. Figure~\ref{fig:tau} shows that our rule-of-thumb works well since within its indicated range, different values of $\tau$ have very little effects on the convergence speeds of both our online algorithms. Next, we consider other values of the latent dimension $K$. In particular, we consider $K=25$ and $K=100$. Figure~\ref{fig:K} shows that the convergence speeds of both our online and batch algorithms are insensitive to this parameter. Then we vary $\rho$ (in both BADMM and OADMM) from 1 to 1000 on a log-scale. We can make two observations from Figure~\ref{fig:rho}. First, as $\rho$ increases,  both ADMM-based algorithms converge slower.  However, the change of the convergence speed of OADMM is much less drastic compared to BADMM. In fact, in less than 10s, all the OADMM algorithms (with different values of $\rho$) converge. This is because for OADMM, a large $\rho$ only has a significant effect on its convergence speed in the initial transient phase. After the algorithm reaches its steady state (i.e., the basis matrix becomes stable), the effect  of a large $\rho$ becomes minimal in terms of solving both \eqref{eq:min_hr} and \eqref{eq:min_W_tr}. Similar effects of the different values of $\kappa$ on the convergence speeds of the PGD-based algorithms are observed in Figure~\ref{fig:kappa}. 
Together, Figure~\ref{fig:rho} and \ref{fig:kappa} reveal {\em another advantage} of our online algorithms. That is, our online algorithms have a significantly larger tolerance of   suboptimal parameters than their batch counterparts. 

\subsubsection{Effects of Proportion of Outliers}
We evaluate the performance of our online (and batch) algorithms (with the canonical parameter setting) on the synthetic dataset with a larger proportion of outliers. Specifically, in generating the synthetic data matrix $\bV$, we keep $F$, $K^o$ and $N$ unchanged, but simultaneously increase $\nu$ and $\tnu$. Figures~\ref{fig:den}(a) and \ref{fig:den}(b) show the convergence results of our algorithms on the synthetic dataset with $(\nu,\tnu)$ equal to $(0.8,0.2)$ and $(0.9,0.3)$ respectively. From Figure~\ref{fig:den}, we observe that our online algorithms still converge fast and steadily (without fluctuations) to a stationary point even though both $\nu$ and $\tnu$ have increased. 

\subsubsection{Experiments on the {\tt CBCL} Dataset}\label{sec:exp_CBCL}
%Figure~\ref{fig:tnu} shows the convergence results of all the algorithms on the synthetic dataset generated with a larger outlier density on each contaminated image, i.e., $\tnu = 0.3$. All the other parameters agree with the canonical setting.  
%Similar to the generation process of the synthetic data, 
To generate the {\tt CBCL} face dataset with outliers, we first replicated it by a factor $p = 50$ (so that the total number of data samples is of the order $10^5$) and randomly permuted the aggregated dataset as described in  Section~\ref{sec:heuristics}. We then contaminated it with outliers in the same manner as for the synthetic dataset. However, we avoided adding observation noise since it had been already introduced by the image acquisition process. 

We then conducted experiments on %the (contaminated) CBCL face 
the contaminated face dataset with the same parameter settings as those on the synthetic data. 
%To generate the contaminated face data, we first vectorized each image in the face dataset and treated it as one column in the clean data matrix. 
%Next we contaminated this data matrix with outliers in the same fashion as done for the synthetic data. 
In the interest of space, we defer the convergence results of all the online and batch algorithms to Section~\ref{sec:exp_res}. %in the supplemental material. 
The results show that both our online algorithms demonstrate fast and steady convergences on this dataset. Moreover, the convergence speeds are insensitive to the key parameters ($\tau$, $K$, $\rho$ and $\kappa$). %These results, together with the convergence results on the synthetic data, show that our online algorithms are able to converge in a fast and steady manner on both synthetic and real data with outliers, without tuning of the parameters. 
Therefore, in the subsequent experiments, we will only focus on  the canonical parameter setting unless mentioned otherwise. 
%thereby demonstrating the wide applicability of our online algorithms. 

\begin{figure}[t]
\subfloat[]{\includegraphics[width=.5\columnwidth,height=.4\columnwidth]{figures/synth_canon_batch.eps}}\hfill
\subfloat[]{\includegraphics[width=.475\columnwidth,height=.4\columnwidth]{figures/synth_canon_OL.eps}}
\caption{The objective values (as a function of time) of (a) our online algorithms and their batch counterparts (b) our online algorithms and other online algorithms. The parameters are set according to the canonical setting.}\label{fig:canon}
\end{figure}

\begin{figure}[t]
\subfloat[OPGD]{\includegraphics[width=.5\columnwidth,height=.38\columnwidth]{figures/synth_OPGD_tau.eps}}\hfill
\subfloat[OADMM]{\includegraphics[width=.475\columnwidth,height=.38\columnwidth]{figures/synth_OADMM_tau.eps}}
\caption{The objective values  of (a) OPGD and (b) OADMM for different values of $\tau$. All the other parameters are set according to the canonical setting.}\label{fig:tau}
\end{figure}


\begin{figure}[t]
\subfloat[ $K=25$ ]{\includegraphics[width=.5\columnwidth,height=.4\columnwidth]{figures/synth_K25_batch.eps}}\hfill
\subfloat[$K=25$ ]{\includegraphics[width=.475\columnwidth,height=.4\columnwidth]{figures/synth_K25_OL.eps}}\\
\subfloat[$K=100$ ]{\includegraphics[width=.5\columnwidth,height=.4\columnwidth]{figures/synth_K100_batch.eps}}\hfill
\subfloat[$K=100$ ]{\includegraphics[width=.475\columnwidth,height=.4\columnwidth]{figures/synth_K100_OL.eps}}
\caption{The objective values (as a function of time) of all the algorithms for different values of $K$. In (a) and (b), $K=25$. In (c) and (d), $K=100$. All the other parameters are set according to the canonical setting.}\label{fig:K}
\end{figure}



\begin{figure}[t]
\subfloat[BADMM]{\includegraphics[width=.5\columnwidth,height=.4\columnwidth]{figures/synth_batch_rho.eps}}\hfill
\subfloat[OADMM]{\includegraphics[width=.475\columnwidth,height=.39\columnwidth]{figures/synth_OL_rho.eps}}
\caption{The objective values  of (a) BADMM and (b) OADMM for different values of $\rho$. All the other parameters are set according to the canonical setting.}\label{fig:rho}
\end{figure}
 
\begin{figure}[t]
\subfloat[BPGD]{\includegraphics[width=.5\columnwidth,height=.4\columnwidth]{figures/synth_batch_eta.eps}}\hfill
\subfloat[OPGD]{\includegraphics[width=.475\columnwidth,height=.4\columnwidth]{figures/synth_OL_eta.eps}}
\caption{The objective values  of (a) BPGD and (b) OPGD for different values of $\kappa$. All the other parameters are set according to the canonical setting.}\label{fig:kappa}
\end{figure}

\begin{figure}[t]
\subfloat[$\nu = 0.8$, $\tnu = 0.2$]{\includegraphics[width=.5\columnwidth,height=.4\columnwidth]{figures/synth_den_batch.eps}}\hfill
\subfloat[$\nu = 0.9$, $\tnu = 0.3$]{\includegraphics[width=.475\columnwidth,height=.39\columnwidth]{figures/synth_den_batch2.eps}}
\caption{The objective values of our online and batch algorithms on the synthetic dataset with a larger proportion of outliers.}\label{fig:den}
%(a) $\nu = 0.8$, $\tnu = 0.2$ (b) $\nu = 0.9$, $\tnu = 0.3$.}
\end{figure}

\subsection{Basis Representations}\label{sec:basis_rep}
We now examine the basis matrices learned by all the algorithms  on the {\tt CBCL} dataset with outliers as introduced in Section~\ref{sec:exp_CBCL}. The parameters controlling the outlier density, $\nu$ and $\tnu$ are set to 0.7 and 0.1 respectively. Figure~\ref{fig:basis} shows the basis representations learned by all the algorithms. From this figure, we observe that the basis images learned by ONMF have large residues of salt and pepper noise. Also, the basis images learned by ORPCA appear to be non-local and do not yield meaningful interpretations. In contrast, the basis images learned by our online (and batch) algorithms are free of noise and much more local than those learned by ORPCA. We can easily observe facial parts from the learned basis. Also, the basis images learned by our four algorithms are of similar quality.
To further enhance the sparsity of the basis images learned, one can add sparsity constraints to each column of the basis matrix, as done for example in \cite{Hoyer_04}. However, we omit such constraints in our discussions. Another parameter affecting the sparsity of the learned basis images is the latent dimension $K$. In general, the larger $K$ is, the sparser the basis images will be. Here we set $K=49$, in accordance with the original setting in \cite{Lee_99}.



\begin{figure}
\subfloat[ONMF]{\includegraphics[width=.475\columnwidth]{figures/ONMF_basis.jpg}}\hfill
\subfloat[ORPCA]{\includegraphics[width=.475\columnwidth]{figures/ORPCA_basis.jpg}}\\
\subfloat[OADMM]{\includegraphics[width=.475\columnwidth]{figures/OADMM_basis.jpg}}\hfill
\subfloat[OPGD]{\includegraphics[width=.475\columnwidth]{figures/OPGD_basis.jpg}}\\
\subfloat[BADMM]{\includegraphics[width=.475\columnwidth]{figures/BADMM_basis.jpg}}\hfill
\subfloat[BPGD]{\includegraphics[width=.475\columnwidth]{figures/BPGD_basis.jpg}}
\caption{Basis representations learned by all the algorithms ($\nu = 0.7$, $\tnu = 0.1$). All the parameters are in the canonical setting ($K=49$).} \label{fig:basis}
\end{figure}





\subsection{Application I: Image Denoising}\label{sec:img_denois}
A natural application that arises from the experiments on the {\tt CBCL} dataset would be image denoising, in particular, removing the salt and pepper noise on images.
%We illustrate an application of our online (and batch) algorithms on removing the salt and pepper noise on images. 
%Our intention here (as well as in Section~\ref{sec:shadow_rmv}) is certainly not to compare our algorithms with the state-of-the-art image processing algorithms. Instead, we aim to demonstrate some possible applications of our online algorithms. 
The metric commonly used to measure the quality of the reconstructed images is peak signal-to-noise ratio (PSNR)\cite{PSNR_98}. A larger value of PSNR indicates better quality of image denoising. With a slight abuse of definition, we apply this metric to all the recovered images. For the batch algorithms, we define %it is   defined as 
\begin{equation}
\mbox{PSNR} \defeq -10\log_{10}\left\{\normt{\bV^o-\hatbW\hatbH}_F^2/(FN)\right\},
\end{equation}
where $\bV^o$, $\hatbW$ and $\hatbH$ denote the matrix of the clean images, estimated basis matrix and estimated coefficient matrix respectively. For the online algorithms, we instead define PSNR in terms of $\bW_N$ (the final dictionary output by Algorithm~\ref{algo:generic}) and the past statistics $\{\bh_i\}_{i\in[N]}$
\begin{align}
\mbox{PSNR} &\defeq -10\log_{10}\left\{\sum_{i=1}^N \norm{\bv_i^o-\bW_N\bh_i}_2^2/(FN)\right\}\\
&\eqcst -10\log_{10} \hatf_N(\bW_N).
\end{align}
In other words, a low averaged regret  will result in a high PSNR. 
Table~\ref{tab:PSNR_K49} shows the image denoising results of all the algorithms on the {\tt CBCL} face dataset. Here the  settings 1, 2 and 3 represent different densities of the salt and pepper noise. Specifically, these settings correspond to $(\nu,\tnu)$ equal to $(0.7,0.1)$, $(0.8,0.2)$ and $(0.9,0.3)$ respectively. %All the parameters are set to the canonical setting. 
%The result of every algorithm on each setting was 
All the results were obtained using ten random initializations  of $\bW_0$.
From Table~\ref{tab:PSNR_K49}(a), we observe that for all the three settings, in terms of PSNRs, our online algorithms only slightly underperform their batch counterparts, but slightly outperform ORPCA and greatly outperform ONMF. %This shows that compared to the other online algorithms, our online algorithms have better capabilities on removing the salt and pepper noise on images. 
From Table~\ref{tab:PSNR_K49}(b), we observe our online algorithms only have slightly longer running times than ORPCA, but are significantly faster than the rest ones. Thus in terms of the trade-off between the computational efficiency and the quality of image denoising, our online algorithms achieve comparable performances with ORPCA but are much better than the rest algorithms. Also, in terms of PSNRs and running times, no significant differences can be observed between our online algorithms.
%Moreover, increased noise density only has minor effects on the performances of our online algorithms.
Similar results were observed when $K = 25$ and $K=100$. See Section~\ref{sec:exp_res} %in the supplemental material 
for details. 

%We introduce two different ways of defining the reconstructed data matrix $\hatbV$ for batch and online algorithms. For the batch algorithms, $\hatbV \defeq \hatbW\hatbH$, where $\hatbW$ and $\hatbH$ are the estimated basis and coefficient matrices output by the algorithms. For the online algorithms, since they can only produce $\hatbW$, we define $\hatbV \defeq \hatf_N(\hatbW)$. 

\begin{table}[t!]\centering\footnotesize
\setlength\tabcolsep{2.5pt}
\caption{Average PSNRs (in dB) and running times (in seconds) with standard deviations of all the algorithms on the {\tt CBCL} face dataset with different noise density ($K=49$).}\label{tab:PSNR_K49}
\subfloat[PSNRs]{
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|}
\hline
\quad&Setting 1 & Setting 2 & Setting 3\\\hline
OADMM &  11.37 $\pm$ 0.02 &  11.35 $\pm$ 0.15 &  11.33 $\pm$ 0.11\\ \hline
OPGD &  11.48 $\pm$ 0.10 &  11.47 $\pm$ 0.05 &  11.39 $\pm$ 0.03\\ \hline
BADMM &  11.53 $\pm$ 0.19 &  11.51 $\pm$ 0.10 &  11.48 $\pm$ 0.03\\ \hline
BPGD &  11.56 $\pm$ 0.07 &  11.52 $\pm$ 0.14 &  11.48 $\pm$ 0.05\\ \hline
ONMF &   5.99 $\pm$ 0.12 &   5.97 $\pm$ 0.18 &   5.95 $\pm$ 0.17\\ \hline
ORPCA &  11.25 $\pm$ 0.04 &  11.24 $\pm$ 0.19 &  11.23 $\pm$ 0.05\\ \hline
\end{tabular}
}\\
\subfloat[Running times]{
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|}
\hline
\quad&Setting 1 & Setting 2 & Setting 3\\\hline
OADMM & 416.68 $\pm$ 3.96 & 422.35 $\pm$ 3.28 & 425.83 $\pm$ 4.25\\ \hline
OPGD & 435.32 $\pm$ 4.80 & 449.25 $\pm$ 3.18 & 458.58 $\pm$ 4.67\\ \hline
BADMM & 1000.45 $\pm$ 10.18 & 1190.55 $\pm$ 5.88 & 1245.39 $\pm$ 10.59\\ \hline
BPGD & 1134.89 $\pm$ 11.37 & 1185.84 $\pm$ 9.83 & 1275.48 $\pm$ 9.48\\ \hline
ONMF & 2385.93 $\pm$ 11.15 & 2589.38 $\pm$ 15.57 & 2695.47 $\pm$ 14.15\\ \hline
ORPCA & 368.35 $\pm$ 3.23 & 389.59 $\pm$ 3.49 & 399.85 $\pm$ 4.12\\ \hline
\end{tabular}
}
\end{table}



\subsection{Application II: Shadow Removal}\label{sec:shadow_rmv}
We evaluated the performances of all the online and batch algorithms on removing shadows in the face images in the {\tt YaleB} dataset. It is well-known from the image processing literature that the shadows in an image can be regarded as inherent outliers. Therefore, the shadow removal task serves as another meaningful application of our online (and batch) algorithms.  We first briefly describe the experiment procedure. For each subject in the {\tt YaleB} dataset, we aggregated and randomly permuted his/her images. We set the aggregation factor $p$ to $50$ in consistency with the previous experiments. We then regarded the resulting data as the input to all the algorithms. 
Finally, we reconstructed the images in a similar way as in Section~\ref{sec:img_denois}.
%The parameters are set to the canonical setting. 
The experiments were run using ten random initializations of $\bW_0$. 
Figure~\ref{fig:shadow} shows some (randomly sampled) reconstructed images on subjects No.2 and No.8 (with one initialization of $\bW_0$). From this figure, we observe that overall, our online algorithms perform almost as well as their batch counterparts, except for small  artifacts (e.g., salt noise) in the recovered images by our online algorithms. We also observe that the other two online algorithms are inferior to our online algorithms. Specifically, ORPCA has more prominent artifacts (e.g., salt noise with larger density) in the recovered images. ONMF in most cases either fails to remove shadow or causes large distortions to the original images. 
The results by other initializations of $\bW_0$ are similar to those shown in Figure~\ref{fig:shadow}.
Table~\ref{tab:shadow_time} shows the running times of all algorithms on the images of subject No.2. The average running times of these algorithms on other subjects are similar to those on subject No.2. This table, together with Figure~\ref{fig:shadow}, suggests  that our online algorithms achieve the best trade-off between the computational efficiency and the quality of shadow removal. Also, both of our online algorithms have similar performances on the shadow removal task. 

\begin{table}[t]\centering\footnotesize
\caption{Average running times (in seconds) of all the algorithms on subject No.2.}\label{tab:shadow_time}
\setlength\tabcolsep{1.5pt}
\begin{tabular}{|c|c|c|c|}
\hline
Algorithms & Time (s) & Algorithms & Time (s)\\\hline
ONMF & $436.11\pm 10.85$ & ORPCA & $16.38\pm 2.43$\\\hline
OADMM & $51.12\pm 2.69$ & OPGD & $62.34\pm 2.75$\\\hline
BADMM & $175.52\pm 7.84$ & BPGD & $212.71\pm 6.33$ \\\hline
\end{tabular}
\end{table}


\begin{figure}
\subfloat{\includegraphics[width=\columnwidth]{figures/shadow_subj2.jpg}}\vspace{-0.3cm}\\
\subfloat{\includegraphics[width=\columnwidth]{figures/shadow_subj8.jpg}}\vspace{-0.1cm}\\
\hspace*{0.5cm}(a)\hspace*{.85cm}(b)\hspace*{.85cm}(c)\hspace*{.85cm}(d)\hspace*{.85cm}(e)\hspace*{.9cm}(f)\hspace*{.85cm}(g)
\caption{Results of shadow removal by all the online and batch algorithms on subjects No.2 (upper)  and No.8 (lower) in the {\tt YaleB} face dataset. The labels (a) to (g) denote the original images and results produced by ONMF, ORPCA, OADMM, OPGD, BADMM, BPGD respectively.}\label{fig:shadow}
\end{figure}


\subsection{Application III: Foreground-Background Separation}\label{sec:backgnd}
Finally, we also evaluated the performance of all the online and batch algorithms on the task of foreground-background separation, which is important in video surveillance \cite{i2r_04,Candes_11}. Since the foreground objects in each video frame generally only occupy a small fraction of pixels, they have been considered as outliers in the literature \cite{Candes_11,Netra_14}. Therefore, our online (and batch) algorithms can be applied to estimate the background scene, as well as learn the foreground objects. In this section, we consider two video sequences, {\tt Hall} and {\tt Escalator} from the {\tt i2r} dataset\cite{i2r_04}. Each video sequence consists of $200$ video frames and the resolutions of the frames in {\tt Hall} and {\tt Escalator} are $144\times 176$ and $130\times 160$ respectively.\footnote{For simplicity we converted the color video frames to gray-scale using the built-in Matlab function \texttt{\textbf{rgb2gray}}.} We set $p=10$ for storage space considerations of the batch algorithms. We also repeated the experiments using ten random initializations of $\bW_0$. %on both video sequences. 

The average running times over the ten initializations of $\bW_0$ (with standard deviations) on the two video sequences are shown in Table~\ref{tab:backgnd_time}. We notice that consistent messages are delivered by Table~\ref{tab:backgnd_time} as those in Table~\ref{tab:PSNR_K49}(b) and Table~\ref{tab:shadow_time}. Namely, the running times of our online algorithms are slightly longer than those of ORPCA but greatly shorter than the rest algorithms. Figure~\ref{fig:background} shows some (randomly sampled)  background  scenes and foreground objects separated by each algorithm. For each frame, the background was reconstructed using the methods introduced in Section~\ref{sec:img_denois}, and the foreground was directly recovered from the corresponding column in the (estimated) outlier matrix $\hatbR$. Since $\hatbR$ is absent in the formulation of ONMF, it is estimated using the difference between the original video frames and the recovered background scenes. From Figure~\ref{fig:background}, we observe that on both video sequences, our online algorithms are able to separate the foreground objects from the background fairly successfully,  with unnoticeable residues on both foreground and background. Compared with the other algorithms, the separation results of our online algorithms are comparable to their batch counterparts and slightly better than ORPCA on the {\tt Hall} sequence (with less residues in the recovered foreground). Due to the lack of robustness, the background scenes recovered by ONMF appear to be dark and the foreground objects cannot be separated from the background.
Thus again, in terms of the trade-off between the computational efficiency and the quality of foreground-background separation, our online algorithms achieve the best performances. Also, both of our online algorithms have similar performances on the foreground-background separation task.



\begin{table}[t]
\centering\footnotesize
\caption{Average running times (in seconds) of all the algorithms on video sequences (a) {\tt Hall} and (b) {\tt Escalator}.}\label{tab:backgnd_time}
\setlength\tabcolsep{1.5pt}
\subfloat[]{\begin{tabular}{|c|c|c|c|}\hline
Algorithms & Time (s) & Algorithms & Time (s)\\\hline
ONMF & $1525.57\pm 14.43$ & ORPCA & $166.85\pm 6.09$\\\hline
OADMM & $172.29\pm 5.64$ & OPGD & $178.46\pm 7.57$\\\hline
BADMM & $1280.06\pm 13.57$ & BPGD & $1167.95\pm 13.38$ \\\hline
\end{tabular}
}\\
\subfloat[]{\begin{tabular}{|c|c|c|c|}\hline
Algorithms & Time (s) & Algorithms & Time (s)\\\hline
ONMF & $1324.86\pm 11.45$ & ORPCA & $160.85\pm 5.03$\\\hline
OADMM & $166.83\pm 5.64$ & OPGD & $170.46\pm 5.91$\\\hline
BADMM & $898.47\pm 8.57$ & BPGD & $867.41\pm 9.35$ \\\hline
\end{tabular}
}
\end{table}


\begin{figure*}
\subfloat{\includegraphics[width=\textwidth,height=.36\textwidth]{figures/hall.eps}}\vspace{-.1cm}\\
\subfloat{\includegraphics[width=\textwidth,height=.36\textwidth]{figures/escalator.eps}}\vspace{-.1cm}\\
$|\hspace{-.15cm}\xleftarrow{\makebox[.16cm]{}}\mbox{(a)}\xrightarrow{\makebox[.16cm]{}}\hspace{-.15cm}$
$|\hspace{-.15cm}\xleftarrow{\makebox[.87cm]{}}\mbox{(b)}\xrightarrow{\makebox[.87cm]{}}\hspace{-.15cm}|$
$\hspace{-.15cm}\xleftarrow{\makebox[.87cm]{}}\mbox{(c)}\xrightarrow{\makebox[.87cm]{}}\hspace{-.15cm}|$
$\hspace{-.15cm}\xleftarrow{\makebox[.87cm]{}}\mbox{(d)}\xrightarrow{\makebox[.87cm]{}}\hspace{-.15cm}|$
$\hspace{-.15cm}\xleftarrow{\makebox[.87cm]{}}\mbox{(e)}\xrightarrow{\makebox[.87cm]{}}\hspace{-.15cm}|$
$\hspace{-.15cm}\xleftarrow{\makebox[.87cm]{}}\mbox{(f)}\xrightarrow{\makebox[.87cm]{}}\hspace{-.15cm}|$
$\hspace{-.15cm}\xleftarrow{\makebox[.87cm]{}}\mbox{(g)}\xrightarrow{\makebox[.87cm]{}}\hspace{-.15cm}|$
\caption{Results of background-foreground separation by all the online and batch algorithms on the {\tt Hall} (upper) and {\tt Escalator} (lower) video sequences. The labels (a) to (g) denote the original video frames and results produced by ONMF, ORPCA, OADMM, OPGD, BADMM, BPGD respectively. For each algorithm, the left column denotes the background and the right column denotes the foreground (moving objects).}\label{fig:background}
\end{figure*}

\section{Conclusion and Future Work} \label{sec:con}
In this paper, we have developed online algorithms for NMF where the data samples are contaminated by outliers. We have shown that the proposed class of algorithms is robust to sparsely corrupted data samples and performs efficiently  on large datasets. %We have also demonstrated the utility of the proposed online algorithms on large datasets. 
Finally, we have proved almost sure convergence guarantees of the objective function as well as the sequence of basis matrices generated by the algorithms. %We believe this methodology with be particularly useful for understanding the latent components of large datasets. 

 We hope to pursue the following three directions for further research. First, it would be interesting to extend the convergence analyses given i.i.d.\ data sequences to   weakly dependent data sequences (e.g.,   martingale  or auto-regressive processes). This is because real data (e.g., video and audio recordings) have correlations among adjacent data samples. %Thus the weak dependence can more realistically model such data sequences.
 Second, it would be meaningful to consider the case where the fit-to-data term between $\bv$ and $\bW\bh+\br$, $d(\bv\Vert \bW\bh+\br)$ is not the squared $\ell_2$ loss,  e.g., the $\beta$-divergence \cite{Fev_09,Fev_11}. This is because in many scenarios, the observation noise is not Gaussian \cite{Fev_09} so other types of loss functions need to be used. Finally, as mentioned in Remark \ref{rmk:cons}, it would be meaningful to consider nonconvex constraint sets and/or regularizers on $\bW$, $\bh$ or $\br$. This is because the nonconvex constraints and regularizers often appear in real applications \cite{Bao_15}.
  
\subsubsection*{Acknowledgment}
The authors are grateful for the helpful clarifications and suggestions from Julien Mairal.

%\newpage
\bibliographystyle{IEEEtran}
\bibliography{RNMF_ref,ORNMF_ref,stoc_ref,math_opt,stat_ref,dataset}

\newpage
\onecolumn
\renewcommand{\thedefinition}{S-\arabic{definition}}
\renewcommand{\thelemma}{S-\arabic{lemma}}
\renewcommand{\thecorollary}{S-\arabic{corollary}}
\renewcommand{\theequation}{S-\arabic{equation}}
\renewcommand{\thesection}{S-\arabic{section}}
\setcounter{section}{0}
\setcounter{equation}{0}
\setcounter{lemma}{0}
\setcounter{corollary}{0}
\setcounter{definition}{0}

{\huge\quad\quad\hspace{4cm} Supplemental Material}\\[.1cm]

In this supplemental material, the indices of all the sections, definitions, lemmas and equations are prepended with an `S' to distinguish those in  the main text. The organization of this article is as follows. In section~\ref{sec:deriv_ADMM}, we drive the ADMM algorithms presented in Section~\ref{sec:ADMM}. In Section~\ref{sec:BRNMF_algo}, we extend our two solvers (based on PGD and ADMM) in Section~\ref{sec:algo} % devised for the online NMF problem with outliers 
to the {\em batch} NMF problems with outliers. Then, we provide detailed proofs of the theorems and lemmas in Section~\ref{sec:conv_analysis} in Section~\ref{sec:regularity} to \ref{sec:Lips_b}. The technical lemmas used in the algorithm derivation (Section~\ref{sec:algo}) and the convergence analysis (Section~\ref{sec:conv_analysis}) are shown in Section~\ref{sec:tech_lemma}. Finally, we show additional experiment results in Section~\ref{sec:exp_res} to supplement those in Section~\ref{sec:numericals}. 
The {\em finite} constants $c$, $c_1$ and $c_2$ are used repeatedly in different sections, and their meanings depend on the context. 

\section{Derivation of the ADMM Algorithms in Section~\ref{sec:ADMM}} \label{sec:deriv_ADMM}
\subsection{Algorithms for \eqref{eq:min_hr}}
Minimizing $\bh$ and $\br$ amounts to solving the following two unconstrained problems
\begin{align}
&\min_\bh \quad \frac{1}{2}\norm{\bW\bh+(\br-\bv)}_2^2 + \balpha^T(\bh-\bu) + \frac{\rho_1}{2}\norm{\bh-\bu}_2^2\nn\\
\Longleftrightarrow\quad & \min_\bh \quad \frac{1}{2}\bh^T(\bW^T\bW+\rho_1\bI)\bh + \left(\bW^T(\br-\bv)-\rho_1\bu+\balpha\right)^T\bh\label{eq:ADMM_h}
\end{align}
and
\begin{align}
&\min_\br \quad \frac{1}{2}\norm{(\bv-\bW\bh)-\br}_2^2 + \frac{\trho_2}{2}\norm{\br-\bq}^2_2 + \bbeta^T(\br-\bq) + \lambda\norm{\br}_1\nn\\
\Longleftrightarrow\quad & \min_\br \quad \frac{1+\trho_2}{2} \br^T\br + (\bbeta-\bv+\bW\bh-\trho_2\bq)^T\br + \lambda\norm{\br}_1\nn\\
\Longleftrightarrow\quad & \min_\br \quad \frac{1+\trho_2}{2} \norm{\br - \frac{\trho_2\bq+\bv-\bbeta-\bW\bh}{1+\trho_2}}_2^2+\lambda\norm{\br}_1.\label{eq:ADMM_r}
\end{align}
We notice that \eqref{eq:ADMM_h} is a standard strongly convex quadratic minimization problem and \eqref{eq:ADMM_r} is a standard proximal minimization problem with $\ell_1$ norm, thus the closed-form optimal solutions for \eqref{eq:ADMM_h} and \eqref{eq:ADMM_r} are 
\begin{align*}
\bh^* &= (\bW^T\bW+\rho_1\bI)^{-1}(\bW^T(\bv-\br)+\rho_1\bu-\balpha)\\
\br^* &= \calS_{\lambda/(1+\trho_2)}\left(\frac{\trho_2\bq+\bv-\bbeta-\bW\bh}{1+\trho_2}\right) = \frac{\calS_\lambda(\trho_2\bq+\bv-\bbeta-\bW\bh)}{1+\trho_2}.
\end{align*}

Minimizing $\bu$ and $\bq$ amounts to solving the following two constrained problems
\begin{align}
\min_{\bu\ge 0} \quad &\frac{\rho_1}{2}\norm{\bh-\bu}_2^2 - \balpha^T\bu\\
\min_{\norm{\bq}_\infty\le M} \quad &\frac{\trho_2}{2}\norm{\br-\bq}_2^2 -\bbeta^T\bq.
\end{align}
Since both constraints  $\bu\ge 0$ and $\norm{\bq}_\infty\le M$ are separable across coordinates, we can simply solve the unconstrained  quadratic minimization problems and then project the optimal solutions to the feasible sets. 

\subsection{Algorithms for \eqref{eq:min_W_tr}}
Minimizing $\bW$ amounts to solving the unconstrained quadratic minimization problem
\begin{align*}
&\min_\bW \quad  \frac{1}{2}\tr\left(\bW^T\bW\bA_t\right) - \tr\left(\bW^T\bB_t\right) + \lrangle{\bD}{\bW-\bQ} + \frac{\trho_3}{2}\norm{\bW-\bQ}_F^2\\
\Longleftrightarrow\quad &\min_\bW \quad \frac{1}{2}\tr\left(\bW\left(\bA_t+\trho_3\bI\right)\bW^T\right) - \tr\left(\left(\bB_t-\bD+\trho_3\bQ\right)^T\bW\right)
\end{align*}
so 
\begin{equation}
\bW^* = \left(\bB_t-\bD+\trho_3\bQ\right)\left(\bA_t+\trho_3\bI\right)^{-1}.
\end{equation}
Minimizing $\bQ$ amounts to solving the constrained quadratic minimization problem
\begin{align*}
&\min_{\bQ\in\calC} \quad \frac{\trho_3}{2}\norm{\bW-\bQ}_F^2 - \lrangle{\bD}{\bQ}\\
\Longleftrightarrow\quad &\min_{\bQ\in\calC} \quad \frac{\trho_3}{2}\norm{\bQ - (\bW+\bD/\trho_3)}_F^2.
\end{align*}
Then we have
\begin{equation}
\bQ^* = \calP_\calC (\bW+\bD/\trho_3).
\end{equation}

\section{Extension to the Batch NMF Problem with Outliers}\label{sec:BRNMF_algo}

\subsection{Problem Formulation}
As usual, we denote the data matrix (with outliers) as $\bV$, basis matrix as $\bW$, coefficient matrix as $\bH$ and outlier matrix as $\bR$. The number of total data samples is denoted as $N$. Then, the batch counterpart for the online NMF problem with outliers can be formulated as follows
\begin{align*}
&\min \quad \frac{1}{2}\norm{\bV-\bW\bH-\bR}_F^2 + \lambda\norm{\bR}_{1,1}\\
&\st \quad \bH\in\bbR_+^{K\times N}, \bR\in\tilcalR, \bW\in\calC, \numberthis \label{eq:brnmf}
\end{align*}
where $\tilcalR = \{\bR\in\bbR^{F\times N}\,|\, \abs{r_{i,j}} \le M, \forall\,(i,j)\in[F]\times[N]\}$.

\subsection{Notations}
In the sequel, we overload soft-thresholding operators $\tilcalS_{\lambda,M}$ and $\calS_\lambda$. When these two operators are applied to matrices, each operator denotes entrywise soft-thresholding. The updated variables are denoted with superscripts `$+$'.


\subsection{Batch algorithm based on PGD (BPGD)}
Based on the principle of block coordinate descent, we update $\bH$, $\bR$ and $\bW$ sequentially as follows.
\begin{align}
\bH^+ &:= \calP_+\left(\bH - \eta_1(\bW)\bW^T(\bW\bH+\bR-\bV)\right)\\
\bR^+ &:= \tilcalS_{\lambda,M} \left(\bV-\bW\bH^+\right)\\
\bW^+_{:j} &:= \frac{\calP_+(\bW-\eta_2(\bH^+)\bG)_{:j}}{\max\{1,\norm{\calP_+(\bW-\eta_2(\bH^+)\bG)_{:j}}_2\}}, \;\forall\,j\in [K],
\end{align}
where $\bG = (\bW\bH^++\bR^+-\bV){\bH^+}^T$, $\eta_1(\bW) \in \left(0,\norm{\bW}_2^{-2}\right]$ and $\eta_2(\bH^+)\in\left(0,\norm{\bH^+{\bH^+}^T}_F^{-1}\right]$.


\subsection{Batch algorithm based on ADMM (BADMM)}
\eqref{eq:brnmf} can be reformulated as 
\begin{align*}
&\min \quad \frac{1}{2}\norm{\bV-\bW\bH-\bR}_F^2 + \lambda\norm{\bR}_{1,1}\\
&\st \quad \bH = \bU,  \bR=\bQ, \bW = \bPsi,  \bU\in\bbR_+^{K\times N}, \bQ\in\tilcalR, \bPsi\in\calC. %\numberthis 
\end{align*}
Thus the augmented Lagrangian is 
\begin{align*}
&\tilcalL(\bH,\bR,\bW,\bU,\bQ,\bPsi,\bA,\bB,\bPsi) = \frac{1}{2}\norm{\bV-\bW\bH-\bR}_F^2 + \lambda\norm{\bR}_{1,1} + \lrangle{\bA}{\bH-\bU} + \lrangle{\bB}{\bR-\bQ} +\lrangle{\bD}{\bW-\bPsi} \\
&\hspace{8.5cm}+\frac{\trho_1}{2}\norm{\bH-\bU}_F^2 + \frac{\trho_2}{2}\norm{\bR-\bQ}_F^2 + \frac{\trho_3}{2} \norm{\bW-\bPsi}_F^2, \numberthis
\end{align*}
where $\bA$, $\bB$ and $\bD$ are dual variables and $\trho_1$, $\trho_2$ and $\trho_3$ are positive penalty parameters. 
Therefore we can derive the following update rules 
\begin{align}
\bH^+ &:= (\bW^T\bW+\trho_1\bI)^{-1}\left(\bW^T(\bV-\bR) + \trho_1\bU-\bA\right)\\
\bR^+ &:= \calS_\lambda(\trho_2\bQ+\bV-\bB-\bW\bH^+)/(1+\trho_2)\\
\bW^+ &:= \left((\bV-\bR^+){\bH^+}^T-\bD+\trho_3\bPsi\right)\left(\bH^+{\bH^+}^T+\trho_3\bI\right)^{-1}\\
\bU^+ &:= \calP_+\left(\bH^+ +\bA/\trho_1\right)\\
\bQ^+ &:= \calP_{\tilcalR} \left(\bR^+ +\bB/\trho_2\right)\\
\bPsi^+ &:= \calP_\calC\left(\bW^++\bD/\trho_3\right)\\
\bA^+ &:= \bA + \trho_1(\bH^+-\bU^+)\\
\bB^+ &:= \bB + \trho_2(\bR^+-\bQ^+)\\
\bD^+ &:= \bD + \trho_3\left(\bW^+ -\bPsi^+\right),
\end{align}



\section{Proof of Lemma~\ref{lem:regularity}} \label{sec:regularity}
Before proving Lemma~\ref{lem:regularity}, we present two lemmas which will be used in the proof. Both lemmas can be proved using straightforward calculations. See Section~\ref{sec:Lips_nablaf} and \ref{sec:Lips_b} for detailed proofs. 

\begin{lemma}\label{lem:Lips_nablaf}
If for each $\bv\in\calV$, both $\bh^*(\bv,\bW)$  and $\br^*(\bv,\bW)$ in \eqref{eq:nablaW_ell} are Lipschitz on $\calC$, with Lipschitz constants $c_1$ and $c_2$ (independent of $\bv$) respectively, then $\bW\mapsto\nabla_\bW\ell(\bv,\bW)$ is Lipschitz on $\calC$ with Lipschitz constant $c_3$ (independent of $\bv$). Consequently, $\nabla f(\bW)$ in \eqref{eq:nablaW_f} is Lipschitz on $\calC$ with Lipschitz constant $c_3$. %Here $c_1$, $c_2$ and $c_3$ are positive universal constants.
\end{lemma}

\begin{lemma} \label{lem:Lips_b}
Let $\bz$, $\bz' \in\calZ\subseteq \bbR^m$ and $\bA$, $\bA'\in\calA\subseteq \bbR^{m\times n}$, where both $\calZ$ and $\calA$ are compact sets. Let $\calB$ be a compact set in $\bbR^n$, and define $g:\calB\to\bbR$ as $g(\bb) = 1/2\norm{\bz-\bA\bb}_2^2 - 1/2 \norm{\bz'-\bA'\bb}^2_2$. Then $g$ is Lipschitz on $\calB$ with Lipschitz constant $c_1\norm{\bz-\bz'}_2 + c_2\norm{\bA-\bA'}_2$, where $c_1$ and $c_2$ are two positive constants. In particular, when both $\bz'$ and $\bA'$ are zero, we have that $\tilg(\bb) = 1/2\norm{\bz-\bA\bb}_2^2$ is Lipschitz on $\calB$ with Lipschitz constant $c$ independent of $\bz$ and $\bA$. %By symmetry, we also have similar results for $h(\bA) = \norm{\bz-\bA\bb}_2^2 - \norm{\bz'-\bA\bb'}^2_2$, where $\bb,\bb'\in\calB$. 
\end{lemma}


It is easy to verify that the following conditions hold
\begin{enumerate}[1)]
\item $\tilell(\bv,\bW, \bh, \br)$ is differentiable on $\calV\times\calC$, for each $(\bh,\br)\in\calH\times\calR$,
\item $\tilell(\bv,\bW, \bh, \br)$ and $\nabla_{(\bv,\bW)}\tilell(\bv,\bW, \bh, \br)$ are continuous on $\calV\times\calC\times\calH\times\calR$,
\item \eqref{eq:new_ithloss} has unique minimizer $(\bh^*(\bv,\bW),\br^*(\bv,\bW))$ for each $(\bv,\bW)\in\calV\times\calC$, due to Assumption~\ref{assum:sc_hr}. 
\end{enumerate}
Thus, we can invoke Danskin's theorem (see Lemma~\ref{lem:min_diff}) to conclude that $\ell(\bv,\bW)$ is differentiable on $\calV\times\calC$ and 
\begin{equation}
\nabla_\bW\ell(\bv,\bW) = \left(\bW\bh^*(\bv,\bW)+\br^*(\bv,\bW)-\bv\right)\bh^*(\bv,\bW)^T. \label{eq:nablaW_ell}
\end{equation}
Furthermore, we can show $(\bh^*(\bv,\bW),\br^*(\bv,\bW))$ is continuous on $\calV\times\calC$ by the maximum theorem (see Lemma~\ref{lem:min_cont}), since the conditions in this theorem are trivially satisfied in our case. Thus, $\nabla_\bW\ell(\bv,\bW)$ is continuous on $\calV\times \calC$. %Consequently, there exists a universal constant $c>0$ such that $\forall\,\bv\in\calV$, $\norm{\nabla_\bW\ell(\bv,\bW)}_F\le c$.  

Leveraging the regularity of $\ell(\bv,\bW)$, we proceed to show the regularity of $f(\bW)$. Since for all $\bv\in\calV$, both $\ell(\bv,\bW)$ and $\nabla_\bW\ell(\bv,\bW)$  are continuous on $\calC$, by Leibniz integral rule (see Lemma~\ref{lem:Leib_int}), we conclude that $f(\bW)$ is differentiable on $\calC$ and 
\begin{equation}
\nabla f(\bW) = \bbE_\bv[\nabla_\bW \ell(\bv,\bW)]. \label{eq:nablaW_f}
\end{equation}

%To show both $\bW\mapsto\nabla_\bW\ell(\bv,\bW)$ and $\nabla f(\bW)$ are uniformly Lipschitz on $\calC$, we make use of the following lemma. This lemma can be proved by straightforward calculations, which are shown in Section~\ref{sec:Lips_nablaf}.


By Lemma~\ref{lem:Lips_nablaf}, to show both $\bW\mapsto\nabla_\bW\ell(\bv,\bW)$ and $\nabla f(\bW)$ are Lipschitz on $\calC$, it suffices to show both $\bh^*(\bv,\bW)$ and $\br^*(\bv,\bW)$ are Lipschitz on $\calC$, for all $\bv\in\calV$. Fix arbitrary $\bv_1,\bv_2\in\calV$ and $\bW_1,\bW_2\in\calC$. Define 
\begin{align*}
d(\bh,\br) &\defeq \tilell(\bv_1,\bW_1,\bh,\br) - \tilell(\bv_2,\bW_2,\bh,\br)\\
& =\frac{1}{2}\norm{\bv_1-\bY_1\bb(\bh,\br)}_2^2 - \frac{1}{2}\norm{\bv_2-\bY_2\bb(\bh,\br)}_2^2
\end{align*}
where $\bY_i = [\bW_i\;\; \bI]$, $i=1,2$ and $\bb(\bh,\br) = [\bh^T\;\;\br^T]^T$. By Lemma~\ref{lem:Lips_b}, we have for all $(\bh_1,\br_1),(\bh_2,\br_2)\in\calH\times\calR$,
\begin{align}
\abs{d(\bh_1,\br_1)-d(\bh_2,\br_2)} &\le \left(c_1\norm{\bv_1-\bv_2}_2 + c_2\norm{\bY_1-\bY_2}_2\right)\norm{\bb(\bh_1,\br_1)-\bb(\bh_2,\br_2)}_2
%& \le  \left(c_1\norm{\bv_1-\bv_2}_2 + c_2\norm{\bW_1-\bW_2}_2\right)\left(\norm{\bh_1-\bh_2}_2+\norm{\br_1-\br_2}_2\right)
\end{align}
where $c_1$ and $c_2$ are positive constants. In particular, we have
\begin{equation}
\abs{d(\bh^*_1,\br^*_1)-d(\bh^*_2,\br^*_2)} \le \left(c_1\norm{\bv_1-\bv_2}_2 + c_2\norm{\bY_1-\bY_2}_2\right)\norm{\bb(\bh^*_1,\br^*_1)-\bb(\bh^*_2,\br^*_2)}_2 \label{eq:Lips_upperbd}
\end{equation}
where $\bh^*_i=\bh^*(\bv_i,\bW_i)$ and $\br^*_i=\br^*(\bv_i,\bW_i)$, $i=1,2$. On the other hand, by Assumption~\ref{assum:sc_hr},
\begin{align*}
\abs{d(\bh^*_2,\br^*_2) - d(\bh^*_1,\br^*_1)} &=\abs{\tilell(\bv_1,\bW_1,\bh^*_2,\br^*_2) - \tilell(\bv_2,\bW_2,\bh^*_2,\br^*_2) - \tilell(\bv_1,\bW_1,\bh^*_1,\br^*_1) + \tilell(\bv_2,\bW_2,\bh^*_1,\br^*_1)}\\
&= (\tilell(\bv_1,\bW_1,\bh^*_2,\br^*_2)-\tilell(\bv_1,\bW_1,\bh^*_1,\br^*_1)) + (\tilell(\bv_2,\bW_2,\bh^*_1,\br^*_1) - \tilell(\bv_2,\bW_2,\bh^*_2,\br^*_2))\\
&\ge m_1\norm{\bb(\bh^*_1,\br^*_1)-\bb(\bh^*_2,\br^*_2)}_2^2.\numberthis \label{eq:sc_lowerbd}
\end{align*}
Combining \eqref{eq:Lips_upperbd} and \eqref{eq:sc_lowerbd}, we have
\begin{equation}
\max\{\norm{\bh^*_1-\bh^*_2}_2,\norm{\br^*_1-\br^*_2}_2\} \le \norm{\bb(\bh^*_1,\br^*_1)-\bb(\bh^*_2,\br^*_2)}_2 \le c'_1\norm{\bv_1-\bv_2}_2 + c'_2\norm{\bW_1-\bW_2}_2
\end{equation}
where $c'_i = c_i/m_1$, $i=1,2$. This indeed shows both $\bh^*(\bv,\bW)$ and $\br^*(\bv,\bW)$ are Lipschitz on $\calV\times\calC$ since
\begin{equation}
c'_1\norm{\bv_1-\bv_2}_2 + c'_2\norm{\bW_1-\bW_2}_2 \le 2\max(c'_1,c'_2)\norm{[\bv_1\;\;\bW_1]-[\bv_2 \;\;\bW_2]}_F.
\end{equation}
Hence we complete the proof. 
%$d(\bh,\br)$ is uniformly Lipschitz on $\calH\times\calR$ with constant $c_1\norm{\bv_1-\bv_2}_2 + c_2\norm{\bW_1-\bW_2}_2$ for some universal positive constants $c_1$ and $c_2$ (note that $\norm{\bY_1-\bY_2}_2 = \norm{\bW_1-\bW_2}_2$).
%\end{proof}


\section{Proof of Lemma~\ref{lem:rate_W}} \label{sec:rate_W}
%\begin{lemma} 
%In Algorithm~\ref{algo:generic},
%\begin{equation}
%\norm{\bW_{t+1}-\bW_t}_F = O\left(\frac{1}{t}\right) \;\;\as
%\end{equation}
%\end{lemma}

%\begin{proof}
By Assumption~\ref{assum:sc_W}, for all $t\ge 1$, we have 
\begin{equation}
\tilf_t(\bW_{t+1}) - \tilf_t(\bW_t) \ge \frac{m_2}{2}\norm{\bW_{t+1}-\bW_t}_F^2, \label{eq:sc_lowerbdW}
\end{equation}
since 
$\bW_t = \argmin_\bW \tilf_t(\bW)$. On the other hand, 
\begin{align*}
\tilf_t(\bW_{t+1}) - \tilf_t(\bW_t) &= \tilf_{t}(\bW_{t+1}) - \tilf_{t+1}(\bW_{t+1}) + \tilf_{t+1}(\bW_{t+1}) - \tilf_{t+1}(\bW_t) + \tilf_{t+1}(\bW_t) - \tilf_t(\bW_t)\\
&\le \tild_t(\bW_{t+1}) - \tild_t(\bW_t)
\end{align*}
where $\tild_t(\bW) \defeq \tilf_{t}(\bW) - \tilf_{t+1}(\bW)$, for all $\bW\in\calC$. We aim to show $\tild_t$ is Lipschitz on $\calC$ with Lipschitz constant only dependent on $t$. For all $\bW\in\calC$, we have
\begin{align*}
\tild_t(\bW) &= \frac{1}{t}\sum_{i=1}^t\frac{1}{2}\norm{\bv_i-\bW\bh_i-\br_i}_2^2+\lambda\norm{\br_i}_1 - \frac{1}{t+1}\sum_{i=1}^{t+1}\frac{1}{2}\norm{\bv_i-\bW\bh_i-\br_i}_2^2+\lambda\norm{\br_i}_1\\
&= \frac{1}{t(t+1)}\left((t+1)\sum_{i=1}^t\frac{1}{2}\norm{\bv_i-\bW\bh_i-\br_i}_2^2+\lambda\norm{\br_i}_1 - t\sum_{i=1}^{t+1}\frac{1}{2}\norm{\bv_i-\bW\bh_i-\br_i}_2^2+\lambda\norm{\br_i}_1\right)\\
&\eqcst \frac{1}{t(t+1)}\left(\sum_{i=1}^t\frac{1}{2}\norm{\bv_i-\bW\bh_i-\br_i}_2^2 - \frac{t}{2}\norm{\bv_{t+1}-\bW\bh_{t+1}-\br_{t+1}}_2^2\right)\\
&= \frac{1}{t(t+1)}\sum_{i=1}^t \left(\frac{1}{2}\norm{\bv_i-\bW\bh_i-\br_i}_2^2 - \frac{1}{2}\norm{\bv_{t+1}-\bW\bh_{t+1}-\br_{t+1}}_2^2\right),
\end{align*}
where $\eqcst$ denotes equality up to an additive constant (independent of $\bW$). By a reasoning similar to the one for Lemma~\ref{lem:Lips_b}, we can show %By Corollary~\ref{cor:Lips_W}, %for $\bW_{t+1}$ and $\bW_t$, %$\forall\,\bW_1,\bW_2\in\calC$, 
there exist some positive  constants $c_1$ and $c_2$ such that 
\begin{align*}
\abs{\tild_t(\bW_{t+1}) - \tild_t(\bW_t)} &\le \frac{1}{t(t+1)}\sum_{i=1}^t \left(c_1\norm{\bh_i-\bh_{t+1}}_2 + c_2\norm{\bv_i-\bv_{t+1}}_2 + c_2\norm{\br_i-\br_{t+1}}_2\right)\norm{\bW_{t+1} - \bW_t}_2\\
&\le \frac{c_3}{t+1}\norm{\bW_{t+1} - \bW_t}_F, \numberthis \label{eq:Lips_upperbdW}
\end{align*}
where $c_3>0$ is a constant since for all $i\ge 1$, $\bv_i$, $\bh_i$ and $\br_i$ are bounded a.s.. Combining \eqref{eq:sc_lowerbdW} and \eqref{eq:Lips_upperbdW}, we have with probability one, 
\begin{equation}
\norm{\bW_{t+1} - \bW_t}_F \le \frac{c'_3}{t+1},
\end{equation}
where $c'_3 = c_3/m_2$. Hence we complete the proof. 
%\end{proof}



%Using similar proof techniques as in Lemma~\ref{lem:Lips_b}, we have the following corollary.
%\begin{corollary} \label{cor:Lips_W}
%Let $\bz$, $\bz' \in\calZ\subseteq \bbR^m$ and $\bb$, $\bb'\in\calB\subseteq \bbR^{n}$, where both $\calZ$ and $\calB$ are compact sets. Let $\calA$ be a compact set in $\bbR^{m\times n}$, and define $g:\calA\to\bbR$ as $g(\bA) = \norm{\bz-\bA\bb}_2^2 - \norm{\bz'-\bA\bb'}^2_2$. Then $g$ is Lipschitz on $\calA$ with constant $c_1\norm{\bz-\bz'}_2 + c_2\norm{\bb-\bb'}_2$ where $c_1$ and $c_2$ are two positive universal constants. In particular, $\tilg(\bA) = \norm{\bz-\bA\bb}_2^2$ is Lipschitz on $\calA$ with a universal Lipschitz constant $c>0$. 
%\end{corollary}


\section{Proof of Theorem~\ref{thm:as_conv}} \label{sec:as_conv}

%\begin{theorem}\label{thm:as_conv}
%In Algorithm~\ref{algo:generic}, the nonnegative stochastic process $\{\tilf_t(\bW_t)\}_{t\ge 1}$ converges a.s.. Furthermore, $\{f(\bW_t)\}_{t\ge 1}$ converges to the same limit as $\{\tilf_t(\bW_t)\}_{t\ge 1}$ a.s.. 
%\end{theorem}
%\begin{proof}
We prove that $\{\tilf_t(\bW_t)\}_{t\ge 1}$ converges a.s. by the quasi-martingale convergence theorem (see Lemma~\ref{lem:quasi_conv}). Let us first define a filtration $\{\calF_t\}_{t\ge 1}$ where $\calF_t \defeq \sigma \left\{\bv_i,\bW_i,\bh_i,\br_i\right\}_{i\in[t]}$ is 
%\begin{equation}
%\calF_t = \sigma \left\{\bv_i,\bW_i,\bh_i,\br_i\right\}_{i\in[t]}.
%\end{equation}
%In other words, $\calF_t$ is 
the minimal $\sigma$-algebra such that $\left\{\bv_i,\bW_i,\bh_i,\br_i\right\}_{i\in[t]}$ are measurable. Also define 
\begin{equation}
u_t \defeq \tilf_t(\bW_t)\quad\mbox{and}\quad
\delta_t \defeq \begin{cases}
1, & \mbox{if} \;\; \bbE[u_{t+1}-u_t|\calF_t] >0 \\
0, & \mbox{otherwise}
\end{cases},
\end{equation}
then it is easy to see $\{u_t\}_{t\ge 1}$ is adapted to $\{\calF_t\}_{t\ge 1}$.
According to the quasi-martingale convergence theorem, it suffices to show $\sum_{t=1}^\infty \bbE[\delta_t(u_{t+1}-u_t)]<\infty$. 
To bound $\bbE[\delta_t(u_{t+1}-u_t)]$, we decompose $u_{t+1}-u_t$ as
\begin{align*}
u_{t+1} -u_t &= \tilf_{t+1}(\bW_{t+1}) - \tilf_{t+1}(\bW_{t}) + \tilf_{t+1}(\bW_{t})- \tilf_{t}(\bW_{t})\\
&= \tilf_{t+1}(\bW_{t+1}) - \tilf_{t+1}(\bW_{t}) + \frac{1}{t+1} \tilell(\bv_{t+1}, \bh_{t+1}, \br_{t+1}, \bW_t) + \frac{t}{t+1} \tilf_t(\bW_t) - \tilf_{t}(\bW_{t})\\
&=  \tilf_{t+1}(\bW_{t+1}) - \tilf_{t+1}(\bW_{t}) + \frac{\ell(\bv_{t+1},\bW_t) - \tilf_t(\bW_t)}{t+1}\numberthis\label{eq:decomp_u0}\\
&= \underbrace{\tilf_{t+1}(\bW_{t+1}) - \tilf_{t+1}(\bW_{t})}_{\lrang{1}} + \frac{\ell(\bv_{t+1},\bW_t)-f_t(\bW_t)}{t+1} + \underbrace{\frac{f_t(\bW_t)- \tilf_t(\bW_t)}{t+1}}_{\lrang{2}}\numberthis \label{eq:decomp_u}.
\end{align*}
By definition, it is easy to see both $\lrang{1},\lrang{2}\le 0$. In \eqref{eq:decomp_u}, we insert the term $f_t(\bW_t)$ in order to invoke Donsker's theorem (see Lemma~\ref{lem:Donsker}). Thus,  
\begin{align*}
\bbE[u_{t+1} -u_t\,|\,\calF_t] \le \frac{\bbE[\ell(\bv_{t+1},\bW_t)-f_t(\bW_t)\,|\,\calF_t]}{t+1} = \frac{f(\bW_t)-f_t(\bW_t)}{t+1}
\end{align*}
and using the definition of $\delta_t$, 
\begin{equation}
\bbE[\delta_t\bbE[u_{t+1} -u_t\,|\,\calF_t]] \le \frac{\bbE\left[\abs{f(\bW_t)-f_t(\bW_t)}\right]}{t+1} \le \frac{\bbE\left[\norm{f-f_t}_\calC\right]}{t+1} = \frac{\bbE\left[\norm{\sqrt{t}(f-f_t)}_\calC\right]}{\sqrt{t}(t+1)},
\end{equation}
where $\norm{\cdot}_\calC$ denotes the uniform norm on $\calC$. By Lemma~\ref{lem:regularity}, we know for all $\bv\in\calV$, $\ell(\bv,\cdot)$ is Lipschitz on $\calC$ with Lipschitz constant independent of $\bv$.  Thus, by Lemma~\ref{lem:suff_Donsker}, the measurable function class $\{\ell(\cdot,\bW):\bW\in\calC\}$ is $\bbP$-Donsker. Consequently $\bbE\left[\norm{\sqrt{t}(f-f_t)}_\calC\right]$ is bounded by a constant $c>0$. Thus, $\bbE[\delta_t\bbE[u_{t+1} -u_t\,|\,\calF_t]]\le c/t^{3/2}$. Since $\bbE[\delta_t(u_{t+1}-u_t)] = \bbE[\bbE[\delta_t(u_{t+1}-u_t)]\,|\,\calF_t]] = \bbE[\delta_t\bbE[u_{t+1}-u_t\,|\,\calF_t]]$,
we have $\sum_{t=1}^\infty \bbE[\delta_t(u_{t+1} -u_t)]<\infty$. Thus $\{\tilf_t(\bW_t)\}_{t\ge 1}$ converges a.s.. Moreover, by Lemma~\ref{lem:quasi_conv}, we also have $\sum_{t=1}^\infty \bbE[\abs{\bbE[u_{t+1}-u_t|\calF_t]}] < \infty$.


Leveraging this result, we proceed to show the almost sure convergence of $\{f(\bW_t)\}_{t\ge 1}$. By Lemma~\ref{lem:suff_Donsker}, the measurable function class $\{\ell(\cdot,\bW):\bW\in\calC\}$ is also $\bbP$-Glivenko-Cantelli. Thus by Glivenko-Cantelli theorem (see Lemma~\ref{lem:GlivenkoCantelli}), we have $\norm{f_t-f}_\calC\convas 0$.  Hence it suffices to show $\{f_t(\bW_t)\}_{t\ge 1}$ converges a.s.. We show this by proving $f_t(\bW_t)-\tilf_t(\bW_t)\convas 0$. First, from \eqref{eq:decomp_u}, we have
\begin{align*}
\frac{\tilf_t(\bW_t)-f_t(\bW_t)}{t+1}
=\;\;& \bbE\left[\frac{\tilf_t(\bW_t)-f_t(\bW_t)}{t+1}\,\bigg|\,\calF_t\right] \\
=\;\;& \bbE[\tilf_{t+1}(\bW_{t+1}) - \tilf_{t+1}(\bW_{t})\,|\,\calF_t] + \bbE\left[\frac{\ell(\bv_{t+1},\bW_t)-f_t(\bW_t)}{t+1}\,\bigg|\,\calF_t\right] - \bbE[u_{t+1} -u_t\,|\,\calF_t]\\
\le\;\;& \frac{\bbE\left[\ell(\bv_{t+1},\bW_t)\,|\,\calF_t\right]-f_t(\bW_t)}{t+1} - \bbE[u_{t+1} -u_t\,|\,\calF_t]\\
\le\;\;& \frac{f(\bW_t)-f_t(\bW_t)}{t+1} - \bbE[u_{t+1} -u_t\,|\,\calF_t]\\
\le\;\;& \frac{\norm{f-f_t}_\calC}{t+1} - \bbE[u_{t+1} -u_t\,|\,\calF_t].
\end{align*}
Since both $\sum_{t=1}^\infty \frac{\norm{f-f_t}_\calC}{t+1}$ and $\sum_{t=1}^\infty  \abs{\bbE[u_{t+1} -u_t\,|\,\calF_t]}$ converge a.s., we conclude 
$\sum_{t=1}^\infty \frac{\tilf_t(\bW_t)-f_t(\bW_t)}{t+1}$
converges a.s.. Define $b_t \defeq \tilf_t(\bW_t)-f_t(\bW_t)$, we show $\abs{b_{t+1}-b_t} = O(1/t)$ a.s. by proving both $|\tilf_{t+1}(\bW_{t+1}) - \tilf_t(\bW_t)| = O(1/t)$ and $|f_{t+1}(\bW_{t+1}) - f_t(\bW_t)| = O(1/t)$ a.s.. First, from \eqref{eq:decomp_u0} and the Lipschitz continuity of $\tilf_t$ on $\calC$,
\begin{align*}
|\tilf_{t+1}(\bW_{t+1}) - \tilf_t(\bW_t)| &\le |\tilf_{t+1}(\bW_{t+1}) - \tilf_{t+1}(\bW_{t})| + \abs{\frac{\ell(\bv_{t+1},\bW_t) - \tilf_t(\bW_t)}{t+1}}\\
&\le c\norm{\bW_{t+1} - \bW_t}_F + \frac{|\ell(\bv_{t+1},\bW_t)| + |\tilf_t(\bW_t)|}{t+1},
\end{align*}
where $c>0$ is a constant independent of $t$. Since both $|\ell(\bv_{t+1},\bW_t)|$ and $|\tilf_t(\bW_t)|$ are bounded on $\calC$ a.s.\ and $\norm{\bW_{t+1} - \bW_t}_F = O(1/t)$ a.s. (by Lemma~\ref{lem:rate_W}), we have $|\tilf_{t+1}(\bW_{t+1}) - \tilf_t(\bW_t)| = O(1/t)$ a.s.. Similarly, by the Lipschitz continuity of $f_t$ on $\calC$, we also have $|f_{t+1}(\bW_{t+1}) - f_t(\bW_t)| = O(1/t)$ a.s.. Now we invoke Lemma~\ref{lem:conv_nonneg} to conclude 
\begin{equation}
f_t(\bW_t)-\tilf_t(\bW_t)\convas 0. \label{eq:as_conv_diff_supp}
\end{equation}
Since $f_t(\bW_t)-f(\bW_t)\convas 0$, both $\{f(\bW_t)\}_{t\ge 1}$ and $\{\tilf_t(\bW_t)\}_{t\ge 1}$ converge to the same almost sure limit. 
%\end{proof}


\section{Proof of Theorem~\ref{thm:local_min}}\label{sec:local_min}
%First, fix a $\bW\in\calC$, it is easy to see $\tilell(\bv,\bW,\bh,\br)$ is Lipschitz on $\calV\times\calH\times\calR$ with some constant $c>0$ independent of $\bW$
%\begin{align*}
%&\abs{\tilell(\bv,\bW,\bh,\br) - \tilell(\bv',\bW,\bh',\br')}\\
%\le\;\; &\abs{\frac{1}{2}\norm{\bv-\bW\bh-\br}_2^2 - \frac{1}{2}\norm{\bv'-\bW\bh'-\br'}_2^2} + \lambda\abs{\norm{\br}_1-\norm{\br'}_1}\\
%\lea\;\;&c'\left(\norm{\bv-\bv'}_2 + \norm{\bh-\bh'}_2 + \norm{\br-\br'}_2\right) + \lambda\sqrt{F}\norm{\br-\br'}_2\\
%\le\;\;&c\left(\norm{\bv-\bv'}_2 + \norm{\bh-\bh'}_2 + \norm{\br-\br'}_2\right) \numberthis \label{eq:Lips_tilell}
%\end{align*}
%where $c'>0$ is a universal constant, $c=c'+\lambda\sqrt{F}$ and $(a)$ can be proved in a similar fashion as Lemma~\ref{lem:Lips_b}. 


%For ease of exposition we focus on a sample path of $\{\bv_t\}_{t\ge 1}$ such that $\tilf_t(\bW_t) - f_t(\bW_t)\to 0$. By \eqref{eq:as_conv_diff}, %such a sample path always exits and 
%the set of all such sample paths has probability measure one. 
%Since $\tilf_t(\bW_t)-f_t(\bW_t)\convas 0$ (see \eqref{eq:as_conv_diff}), 
By \eqref{eq:as_conv_diff_supp}, it suffices to show that for every realization of $\{\bv_t\}_{t\ge 1}$ such that $\tilf_t(\bW_t) - f_t(\bW_t)\to 0$, each subsequential limit of $\{\bW_t\}_{t\ge 1}$ is a stationary point of $f$. We focus on such a realization, then all the variables in the sequel become deterministic. (With a slight abuse of notations we use the same notations to denote the deterministic variables.) %For any $t\ge 1$, define $\bA_t \defeq \bA_t/t$ and $\bB_t \defeq \bB_t/t$. 
By the compactness of $\calV$, $\calH$ and $\calR$, both sequences $\{\bA_t\}_{t\ge 1}$ and $\{\bB_t\}_{t\ge 1}$ are bounded. Thus there exist compact sets $\calA$ and $\calB$ such that $\{\bA_t\}_{t\ge 1} \subseteq \calA$ and $\{\bB_t\}_{t\ge 1} \subseteq \calB$. Similar reasoning shows that the sequence $\{\tilf_t(\vecz)\}_{t\ge 1}$ resides in a compact set $\tilcalF$. By the compactness of $\calC$, there exists a convergent subsequence $\{\bW_{t_m}\}_{m\ge 1}$ in $\{\bW_t\}_{t\ge 1}$.  Also, by the compactness of $\calA$, $\calB$ and $\tilcalF$, it is possible to find convergent subsequences $\{\bA_{t_k}\}_{k\ge 1}$, $\{\bB_{t_k}\}_{k\ge 1}$ and $\{\tilf_{t_k}(\vecz)\}_{k\ge 1}$ such that $\{t_k\}_{k\ge 1}\subseteq\{t_m\}_{m\ge 1}$. Thus, we focus on the convergent sequences $\{\bW_{t_k}\}_{k\ge 1}$, $\{\bA_{t_k}\}_{k\ge 1}$, $\{\bB_{t_k}\}_{k\ge 1}$ and $\{\tilf_{t_k}(\vecz)\}_{k\ge 1}$ and drop the subscript $k$ to make notations uncluttered. We denote the limits of the sequences $\{\bW_{t}\}_{t\ge 1}$, $\{\bA_{t}\}_{t\ge 1}$ and $\{\bB_{t}\}_{t\ge 1}$ as $\barbW$, $\barbA$ and $\barbB$ respectively. 
%Again by the compactness of $\calV\times\calH\times\calR$, there exists a convergence subsequence $\{(\bv_{t_k},\bh_{t_k},\br_{t_k})\}_{k\ge 1}$ in $\{(\bv_t,\bh_t,\br_t)\}_{t\ge 1}$ such that $\{t_k\}_{k\ge 1}\subseteq\{t_m\}_{m\ge 1}$. Thus, we focus on the sequences $\{\bW_{t_k}\}_{k\ge 1}$ and $\{(\bv_{t_k},\bh_{t_k},\br_{t_k})\}_{k\ge 1}$, and drop the subscript $k$ to make notations uncluttered. Let $\barbW\in\calC$ and $(\barbv,\barbh,\barbr)\in\calV\times\calH\times\calR$ be the limits of $\{\bW_{t}\}_{t\ge 1}$ and $\{(\bv_{t},\bh_{t},\br_{t})\}_{t\ge 1}$ respectively. 


First, we show the sequence of differentiable functions $\{\tilf_t\}_{t\ge 1}$ converges uniformly to a differentiable function $\barf$. Since $\{\tilf_{t}(\vecz)\}_{t\ge 1}$ converges, it suffices to show the sequence $\{\nabla \tilf_t\}_{t\ge 1}$ converges uniformly to a function $\barh$. Since $\nabla \tilf_t(\bW) = \bW\bA_t - \bB_t$, for any $t,t'\ge 1$ and any $\bW\in\calC$, we have
\begin{align*}
\norm{\nabla \tilf_t(\bW) - \nabla \tilf_{t'}(\bW)}_F  &= \norm{\bW\left(\bA_t - \bA_{t'}\right) -  \left(\bB_t - \bB_{t'}\right)}_F \\
&\le \norm{\bW}_F \normt{\bA_t - \bA_{t'}}_F + \normt{\bB_t - \bB_{t'}}_F\\
&\le \sqrt{K}\left(\normt{\bA_t - \bA_{t'}}_F + \normt{\bB_t - \bB_{t'}}_F\right). \numberthis \label{eq:Lips_nablaf_t}
\end{align*}
From \eqref{eq:Lips_nablaf_t}, it is easy to see $\barh(\bW) = \bW\barbA - \barbB$ since
$\sup_{\bW\in\calC} \normt{\nabla \tilf_t(\bW) - \barh(\bW)}_F  \le \sqrt{K}\left(\normt{\bA_t - \barbA}_F + \normt{\bB_t - \barbB}_F\right)$. 

Next, define $g_t \defeq \tilf_t - f_t$, for all $t\ge 1$. By definition, we have $g_t(\bW)\ge 0$, for any $\bW\in \calC$. Since $\tilf_t\convu \barf$ and $f_t \convu f$ (by Glivenko-Cantelli theorem), we have $g_t \convu \barg \defeq \barf - f$. 
Since both $\barf$ and  $f$ are differentiable, $\barg$ is differentiable and $\nabla f = \nabla \barf - \nabla \barg$. To show $\barbW$ is a stationary point of $f$, it suffices to show for any $\bW\in\calC$, the directional derivative $\lrangle{\nabla f(\barbW)}{\bW - \barbW}\ge 0$. We show this by proving  $\lrangle{\nabla \barf(\barbW)}{\bW - \barbW}\ge 0$ and $\lrangle{\nabla \barg(\barbW)}{\bW - \barbW}= 0$ for any $\bW\in\calC$.

By definition, for any $\bW\in\calC$ and $t\ge 1$, we have $\tilf_t(\bW_t)  \le \tilf_t(\bW)$.  First consider 
\begin{align*}
\abs{\tilf_t(\bW_t) - \barf(\barbW)} & = \abs{\tilf_t(\bW_t) - \barf(\bW_t) + \barf(\bW_t) - \barf(\barbW)}\\
&\le \norm{\tilf_t - \barf}_\calC + \abs{\barf(\bW_t) - \barf(\barbW)}.
\end{align*}
Since $\tilf_t\convu \barf$ and $\barf$ is continuous, we have $\tilf_t(\bW_t) \to \barf(\barbW)$ as $t\to\infty$. Thus $\barf(\barbW)\le \barf(\bW)$, for any $\bW\in\calC$. This implies  $\lrangle{\nabla \barf(\barbW)}{\bW - \barbW}\ge 0$. 

Next we show $\lrangle{\nabla \barg(\barbW)}{\bW - \barbW}= 0$ for any $\bW\in\calC$. It suffices to show $\nabla \barg(\barbW) = \vecz$. Since both $\tilf_t$ and $f_t$ are differentiable, $g_t$ is differentiable and $\nabla g_t = \nabla \tilf_t - \nabla f$. First it is easy to see $\nabla g_t$ is Lipschitz  on $\calC$ with constant $L>0$ independent of $t$ since both $\nabla \tilf_t$ and $\nabla f_t$ are Lipschitz on $\calC$ with constants independent of $t$. It is possible to construct another differentiable function $\tilg_t$ with domain $\bbR^{F\times K}$ such that $\tilg_t$ is nonnegative with a $L$-Lipschitz gradient on $\bbR^{F\times K}$ and $\tilg_t (\bW) = g_t(\bW)$ for all $\bW\in\calC$. Thus
\begin{align*}
\frac{1}{2L}\normt{\nabla g_t(\bW_t)}_F^2=\frac{1}{2L}\normt{\nabla \tilg_t(\bW_t)}_F^2 \le \tilg_t(\bW_t) - \tilg_t^* \le g_t(\bW_t) = \tilf_t(\bW_t) - f_t(\bW_t),
\end{align*}
where $\tilg_t^* = \inf_{\bW\in\bbR^{F\times K}} \tilg_t(\bW)\ge 0$. Since  $\tilf_t(\bW_t) - f_t(\bW_t)\to 0$, we have $\nabla g_t(\bW_t)\to \vecz$ as $t\to\infty$. Now consider the first-order Taylor expansion of $g_t$ at $\bW_t$
\begin{equation}
g_t(\bW) = g_t(\bW_t) + \lrangle{\nabla g_t(\bW_t)}{\bW-\bW_t} + o\left(\normt{\bW-\bW_t}_F\right), \;\; \forall\,\bW\in\calC. 
\end{equation}
As $t\to\infty$, we have
\begin{equation}
\barg(\bW) = \barg(\barbW) + o\left(\norm{\bW-\barbW}_F\right), \;\; \forall\,\bW\in\calC. \label{eq:barg_Taylor1}
\end{equation}
On the other hand, we have
\begin{equation}
\barg(\bW) = \barg(\barbW) + \lrangle{\nabla \barg(\barbW)}{\bW-\barbW} + o\left(\norm{\bW-\barbW}_F\right), \;\; \forall\,\bW\in\calC.  \label{eq:barg_Taylor2}
\end{equation}
Comparing \eqref{eq:barg_Taylor1} and \eqref{eq:barg_Taylor2}, we have
\begin{equation}
\lrangle{\nabla \barg(\barbW)}{\bW-\barbW} + o\left(\norm{\bW-\barbW}_F\right) = 0, \;\; \forall\,\bW\in\calC.
\end{equation}
Therefore we conclude $\nabla \barg(\barbW) = \vecz$. 
%Denote the Lipschitz constant of $\nabla g_t$ as $L$. 
%Consider
%\begin{align*}
%\normt{\nabla g_t(\bW)}_F &\le \normt{\nabla \tilf_t(\bW)}_F + \normt{\nabla f(\bW)}_F\\
%&\le \normt{\bW\bA_t}_F + \normt{\bB_t}_F + \frac{1}{t}\sum_{i=1}^t \normt{\nabla_\bW \ell(\bv_i,\bW)}_F.
%\end{align*}
%By the boundedness of $\bW$, $\bA_t$ and $\bB_t$, and the Lipschitz continuity of $\nabla_\bW\ell(\bv_i,\bW)$ on $\calC$ (with Lipschitz constant independent of $\bv$), we conclude that $\normt{\nabla g_t(\bW)}_F$ is bounded on $\calC$. Hence $\nabla g_t$ is Lipschitz on $\calC$. 


%Define $\tilell_t (\bW) \defeq \tilell(\bv_t,\bW,\bh_t,\br_t)$ and $\barell (\bW) \defeq \tilell(\barbv,\bW,\barbh,\barbr)$. By \eqref{eq:Lips_tilell} we have $\tilell_t\convu\barell$ on $\calC$ since
%\begin{align*}
%\normt{\tilell_t-\barell}_\calC = \sup_{\bW\in\calC}\abs{\tilell_t(\bW) - \barell(\bW)} \le c\left(\norm{\bv_t-\barbv}_2 + \norm{\bh_t-\barbh}_2 + \norm{\br_t-\barbr}_2\right) \to 0.
%\end{align*}
%Consequently, we have $\tilf_t\convu\barell$ since
%\begin{align*}
%\norm{\tilf_t-\barell}_\calC = \norm{\frac{1}{t}\sum_{i=1}^t \left(\tilell_i - \barell\right)}_\calC \le \frac{1}{t}\sum_{i=1}^t\norm{\tilell_i - \barell}_\calC
%%\le \frac{1}{t}\sum_{i=1}^t\abs{\tilell(\bv_i,\bW,\bh_i,\br_i) - \tilell(\barbv,\bW,\barbh,\barbr)}\\
%%\le\;\; & \sup_{\bW\in\calC}\frac{1}{t}\sum_{i=1}^t c\left(\norm{\bv_i-\barbv}_2 + \norm{\bh_i-\barbh}_2 + \norm{\br_i-\barbr}_2\right)\\
%\end{align*} 


\section{Proof of Lemma~\ref{lem:Lips_nablaf} } \label{sec:Lips_nablaf}

%\begin{proof}
Since $\bv$, $\bW$, $\bh^*(\bv,\bW)$  and $\br^*(\bv,\bW)$ are all bounded, there exist positive constants $M_1$, $M_2$, $M_3$ and $M_4$ that upper bound $\norm{\bv}$, $\norm{\bW}$, $\norm{\bh^*(\bv,\bW)}$  and $\norm{\br^*(\bv,\bW)}$ respectively. Here the matrix norm is the one induced by the (general) vector norm. Take arbitrary $\bW_1$ and $\bW_2$ in $\calC$, we have 
\begin{align*}
\norm{\nabla f(\bW_1) - \nabla f(\bW_2)} &= \norm{\bbE_\bv[\nabla_{\bW}\ell(\bv,\bW_1) - \nabla_{\bW}\ell(\bv,\bW_2)]}\\
&\le \bbE_\bv \norm{\nabla_{\bW}\ell(\bv,\bW_1) - \nabla_{\bW}\ell(\bv,\bW_2)}.
\end{align*}
Fix an arbitrary $\bv\in\calV$ we have
\begin{align*}
\norm{\nabla_{\bW}\ell(\bv,\bW_1) - \nabla_{\bW}\ell(\bv,\bW_2)} &\le \norm{\bW_1\bh^*(\bv,\bW_1)\bh^*(\bv,\bW_1)^T - \bW_2\bh^*(\bv,\bW_2)\bh^*(\bv,\bW_2)^T} \\
&+ \norm{\br^*(\bv,\bW_1)\bh^*(\bv,\bW_1)^T - \br^*(\bv,\bW_2)\bh^*(\bv,\bW_2)^T} \numberthis \label{eq:norm_nabla_ell}\\
&+ \norm{\bv\bh^*(\bv,\bW_1)^T - \bv\bh^*(\bv,\bW_2)^T}. 
\end{align*}
We bound each term on the RHS of \eqref{eq:norm_nabla_ell} as follows
\begin{flalign*}
&\norm{\bW_1\bh^*(\bv,\bW_1)\bh^*(\bv,\bW_1)^T - \bW_2\bh^*(\bv,\bW_2)\bh^*(\bv,\bW_2)^T}& \\
\le \;&\norm{\bW_1}\norm{\bh^*(\bv,\bW_1)}\norm{\bh^*(\bv,\bW_1)-\bh^*(\bv,\bW_2)} + \norm{\bW_1\bh^*(\bv,\bW_1) - \bW_2\bh^*(\bv,\bW_2)}\norm{\bh^*(\bv,\bW_2)}&\\
\le \;&M_2M_3c_1\norm{\bW_1-\bW_2} + \left(\norm{\bW_1}\norm{\bh^*(\bv,\bW_1) - \bh^*(\bv,\bW_2)} + \norm{\bW_1-\bW_2}\norm{\bh^*(\bv,\bW_2)}\right)\norm{\bh^*(\bv,\bW_2)}&\\
\le \;&M_2M_3c_1\norm{\bW_1-\bW_2} + M_3\left(M_2c_1\norm{\bW_1-\bW_2} + M_3\norm{\bW_1-\bW_2}\right)\\
= \;&(2c_1M_2M_3 + M_3^2)\norm{\bW_1-\bW_2},&
\end{flalign*}
\begin{flalign*}
&\norm{\br^*(\bv,\bW_1)\bh^*(\bv,\bW_1)^T - \br^*(\bv,\bW_2)\bh^*(\bv,\bW_2)^T}&\\
\le\;&\norm{\br^*(\bv,\bW_1)}\norm{\bh^*(\bv,\bW_1)-\bh^*(\bv,\bW_2)} + \norm{\br^*(\bv,\bW_1)-\br^*(\bv,\bW_2)}\norm{\bh^*(\bv,\bW_2)}&\\
\le\;&c_1M_4\norm{\bW_1-\bW_2} + c_2M_3\norm{\bW_1-\bW_2}&\\
\le\;&(c_1M_4 + c_2M_3)\norm{\bW_1-\bW_2},&
\end{flalign*}
and
\begin{flalign*}
&\norm{\bv\bh^*(\bv,\bW_1)^T - \bv\bh^*(\bv,\bW_2)^T}\le c_1M_1\norm{\bW_1-\bW_2}.&
\end{flalign*}
Thus, take $c_3 = 2c_1M_2M_3 + M_3^2 + c_1M_4 + c_2M_3 +  c_1M_1$ and we finish the proof. 


\section{Proof of Lemma~\ref{lem:Lips_b}} \label{sec:Lips_b}
%We first rewrite $g$ in standard quadratic form
%\begin{equation}
%g(\bb) = \bb^T(\bA^T\bA-{\bA'}^T\bA')\bb - 2(\bA^T\bz- {\bA'}^T\bz')^T\bb + \norm{\bz}^2_2 - \norm{\bz'}^2_2.
%\end{equation}
%Thus, $\forall\,\bb,\bb'\in\calB$,
%\begin{align*}
%\abs{g(\bb)-g(\bb')} &= \big|\bb^T(\bA^T\bA-{\bA'}^T\bA')\bb - \bb^T(\bA^T\bA-{\bA'}^T\bA')\bb' + \bb^T(\bA^T\bA-{\bA'}^T\bA')\bb' \\
%&\qquad\qquad\qquad\qquad\qquad - {\bb'}^T(\bA^T\bA-{\bA'}^T\bA')\bb' - 2(\bA^T\bz- {\bA'}^T\bz')^T(\bb-\bb')\big|\\
%&\le \abs{\bb^T(\bA^T\bA-{\bA'}^T\bA')(\bb-\bb')} + \abs{(\bb-\bb')^T(\bA^T\bA-{\bA'}^T\bA')\bb'} + 2\abs{(\bA^T\bz- {\bA'}^T\bz')^T(\bb-\bb')}\\
%&\le \lrpar{\underbrace{\normt{(\bA^T\bA-{\bA'}^T\bA')^T\bb}_2}_{\lrang{1}} + \underbrace{\normt{(\bA^T\bA-{\bA'}^T\bA')\bb'}_2}_{\lrang{2}} + \underbrace{2\normt{\bA^T\bz- {\bA'}^T\bz'}_2}_{\lrang{3}}}\norm{\bb-\bb'}_2
%\end{align*}
It suffices to show $\norm{\nabla g(\bb)}_2 \le c_1\norm{\bz-\bz'}_2 + c_2\norm{\bA-\bA'}_2$ for any $\bb\in\calB$ and some positive constants $c_1$ and $c_2$ (independent of $\bb$). We write  $\norm{\nabla g(\bb)}_2$ as
\begin{align*}
\norm{\nabla g(\bb)}_2 &= \normt{{\bA'}^T\left(\bA'\bb - \bz'\right) - \bA^T\left(\bA\bb-\bz\right)}_2\\
& = \normt{({\bA'}^T\bA'-\bA^T\bA)\bb - ({\bA'}^T\bz' - \bA^T\bz)}_2\\
& \le \underbrace{\normt{{\bA'}^T\bA'-\bA^T\bA}_2\norm{\bb}_2}_{\lrang{1}} + \underbrace{\normt{{\bA'}^T\bz' - \bA^T\bz}_2}_{\lrang{2}}.
\end{align*}
By the  compactness of $\calZ$, $\calA$ and $\calB$, there exist positive  constants $M_1$, $M_2$ and $M_3$ such that $\norm{\bz}_2\le M_1$, $\norm{\bA}_2\le M_2$ and $\norm{\bb}_2\le M_3$, for any $\bz\in\calZ$, $\bA\in\calA$ and $\bb\in\calB$. Thus, 
\begin{align*}
\lrang{1} &\le M_3\normt{\bA^T\bA-\bA^T\bA'+\bA^T\bA'-{\bA'}^T\bA'}_2\\
&\le  M_3\lrpar{\normt{\bA^T(\bA-\bA')}_2+\norm{(\bA-\bA')^T\bA'}_2}\\
&\le 2M_2M_3\norm{\bA-\bA'}_2.
\end{align*}
%Similarly, we also have $\lrang{2}\le 2M_2M_3\norm{\bA-\bA'}_2$. 
Similarly for $\lrang{2}$ we have
\begin{align*}
\lrang{2} &= \norm{\bA^T\bz- \bA^T\bz'+\bA^T\bz'-{\bA'}^T\bz'}_2\\
&\le \norm{\bA^T(\bz-\bz')}_2+\norm{(\bA-\bA')^T\bz'}_2\\
&\le M_2\norm{\bz-\bz'}_2 + M_1\norm{\bA-\bA'}_2.
\end{align*}
Hence $\lrang{1}+\lrang{2}\le M_2\norm{\bz-\bz'}_2 + (M_1+2M_2M_3)\norm{\bA-\bA'}_2$. We now take $c_1 = M_2$ and $c_2 = M_1+2M_2M_3$ to complete the proof. 

\section{Technical lemmas}\label{sec:tech_lemma}

\begin{lemma}[{\cite[Lemma 5]{ZhangBox_15}}] \label{lem:L1_box}
Let $I$ be a closed interval in $\bbR$. Define $g_{\tau,I}(t) = \tau\abs{t} + \delta_I(t)$, where $\delta_I$ is the indicator function for the interval $I$. Then the proximal operator for $g_{\tau,I}$ is given by
\begin{equation}
\prox_{g_{\tau,I}} (q) = \Pi_I(\calS_\tau(q)),
\end{equation}
where $q\in\bbR$, $\calS_\tau$ is the soft-thresholding operator with threshold $\tau$ and $\Pi_I$ is the Euclidean projector onto the interval $I$.
\end{lemma}

\begin{lemma}[Projection onto nonnegative $\ell_2$ balls] \label{lem:proj_nonnegL2}
Let $\calC' \defeq \{\bx\in\bbR_+^n\,|\,\norm{\bx}\le 1\}$. Then for all $\by\in\bbR^n$, 
\begin{equation}
\Pi_{\calC'}(\by) = \frac{\by_+}{\max\left\{1,\norm{\by_+}_2\right\}},
\end{equation}
where $(\by_+)_i = \max\{0,y_i\}$, $\forall\,i\in[n]$. 
\end{lemma}
\begin{proof}
The KKT conditions for 
\begin{align*}
&\min \quad \norm{\by-\bx}_2^2\\
&\st \quad  \bx\ge 0, \;\norm{\bx}_2\le 1
\end{align*}
are given by
\begin{align}
&\bx^*\ge 0, \;\norm{\bx^*}_2\le 1, \; \lambda^*\ge 0 \label{eq:KKT_feas1}\\
&(\lambda^*+1)\bx^*\ge \by, \label{eq:KKT_feas2}\\
&\lambda^*(\norm{\bx^*}_2^2-1) = 0, \label{eq:KKT_comp1}\\
&(\lambda^*+1)(x^*_i)^2 =x^*_iy_i, \;\forall\,i\in[n]. \label{eq:KKT_comp2}
\end{align}
We fix an $i\in[n]$. Define $\calI \defeq \{i\in[n]\,|\,y_i>0\}$. For any $\bz\in\bbR^n$, let $\bz_\calI$ to be the subvector of $\bz$ with indices from $\calI$. If $y_i\le 0$, then by \eqref{eq:KKT_comp2} $x^*_i=0$. If $y_i>0$, then by \eqref{eq:KKT_feas2} $x^*_i>0$. Thus by \eqref{eq:KKT_comp2} we have $x^*_i = y_i/(\lambda^*+1)$. If $\lambda^*=0$, then $x^*_i = y_i$. In this case $\norm{\by_\calI}_2 = \norm{\bx^*_\calI}_2  = \norm{\bx^*}_2 \le 1$. If $\lambda^*>0$, then by \eqref{eq:KKT_comp1} $\norm{\bx^*}_2^2=1$. Then $\norm{\bx^*}_2^2 = \norm{\bx^*_\calI}_2^2 = \norm{\by_\calI}_2^2/(\lambda^*+1)^2 = 1$. This means $\lambda^*+1 = \norm{\by_\calI}_2$ so $x^*_i = y_i/\norm{\by_\calI}_2$. Also, we notice in such case $y_i>x^*_i>0$ so $\norm{\by_\calI}_2>1$. Combining both cases where $\lambda^*=0$ and $\lambda^*>0$, we have
$x_i^* = y_i/\max\{1,\norm{\by_\calI}_2\}$, for all $i\in\calI$.
%\begin{equation}
%\Pi_\calC(\by) = \argmin_{\bx\ge 0, \;\norm{\bx}_2\le 1} \norm{\by-\bx}_2^2, \label{eq:nonnegProj_l2}
%\end{equation}
%we solve the 
\end{proof}

\begin{lemma}[Danskin's Theorem; {\cite[Theorem 4.1]{Bon_98}}] \label{lem:min_diff}
Let $\calX$ be a metric space and $\calU$ be a normed vector space. Let $f:\calX\times \calU\to\bbR$ have the following properties
\begin{enumerate}
\item $f(x,\cdot)$ is differentiable on $\calU$, for any $x\in\calX$.
\item $f(x,u)$ and $\nabla_u f(x,u)$ are continuous on $\calX\times\calU$.
\end{enumerate}
Let $\Phi$ be a compact set in $\calX$. Define $v(u) = \inf_{x\in\Phi}f(x,u)$ and $S(u) = \argmin_{x\in\Phi}f(x,u)$, then $v(u)$ is directionally differentiable and its directional derivative along $d\in\calU$, $v'(u,d)$ is given by
\begin{equation}
v'(u,d) = \min_{x\in S(u)} \nabla_uf(x,u)^Td.
\end{equation}
In particular, if for some $u_0\in\calU$, $S(u_0) = \{x_0\}$, then $v$ is differentiable at $u=u_0$ and $\nabla v(u_0) = \nabla_uf(x_0,u_0)$.
\end{lemma}

\begin{lemma}[The Maximum Theorem; {\cite[Theorem 14.2.1 \& Example 2]{Syd_05}}]\label{lem:min_cont}
Let $\calP$ and $\calX$ be two metric spaces. Consider a maximization problem
\begin{equation}
\max_{x\in B(p)} f(p,x), \label{eq:max_problem}
\end{equation}
where $B:\calP \twoheadrightarrow \calX$ is a correspondence and  $f:\calP\times \calX\to\bbR$ is a function. If $B$ is compact-valued and continuous on $\calP$ and $f$ is continuous on $\calP\times \calX$, then the correspondence $S(p) = \argmax_{x\in B(p)} f(p,x)$ is compact-valued and upper hemicontinuous, for any $p\in\calP$.
%\begin{enumerate}
%\item The correspondence $S(p) = \argmax_{x\in B(p)} f(p,x)$ is compact-valued and upper hemicontinuous, for any $p\in\calP$.
%\item The function $v(p) = \max_{x\in B(p)} f(p,x)$ is continuous on $\calP$. 
%\end{enumerate} 
In particular, if for some $p_0\in\calP$, $S(p_0)=\{s(p_0)\}$, where $s:\calP\to\calX$ is a function, then $s$ is continuous at $p=p_0$. Moreover, we have the same conclusions if the maximization in \eqref{eq:max_problem} is replaced by minimization. 
\end{lemma}


\begin{lemma}[Leibniz Integral Rule] \label{lem:Leib_int}
Let $\calX$ be an open set in $\bbR^n$ and let $(\Omega,\calA,\mu)$ be a measure space. If $f:\calX\times\Omega\to\bbR$ satisfies
\begin{enumerate}
\item For all $x\in\calX$, the mapping $\omega\mapsto f(x,\omega)$ is Lebesgue integrable. % on $\Omega$.
\item For all $\omega\in\Omega$, $\nabla_xf(x,\omega)$ exists on $\calX$.
%\item There exists $\Omega'\subseteq\Omega$ such that $\mu(\Omega')=1$ and $\forall\,\omega\in\Omega'$, $\nabla_xf(x,\omega)$ exists on $\calX$.
%\item For almost all $\omega\in\Omega$ and for all $x\in\calX$, there exists a nonnegative Lebesgue integrable function $g:\Omega\to\bbR_+$ such that $|\nabla_xf(x,\omega)|\le g(\omega)$.
\item For all $x\in\calX$,  the mapping $\omega\mapsto \nabla_x f(x,\omega)$ is Lebesgue integrable. % on $\Omega'$.
\end{enumerate}
Then $\int_\Omega f(x,\omega) \, d\mu(\omega)$ is differentiable and  
\begin{equation}
\nabla_x \int_\Omega f(x,\omega) \, d\mu(\omega) = \int_\Omega \nabla_x f(x,\omega) \, d\mu(\omega).
\end{equation}
\end{lemma}
\begin{remark}
This is a simplified version of the Leibniz Integral Rule. See \cite[Theorem 16.8]{Bill_86} for weaker conditions on $f$.
\end{remark}


\begin{definition}[Quasi-martingale; {\cite[Definition 1.4]{Fisk_65}}]\label{def:quasi_mart}
Let $\{X_t\}_{t\in\calT}$ be a stochastic process and %defined on $(\Omega,\calF,\bbP)$ 
let $\{\calF_t\}_{t\in\calT}$ be the filtration to which %the stochastic process 
$\{X_t\}_{t\in\calT}$ is adapted, where $\calT$ is a subset of the real line. We call $\{X_t\}_{t\in\calT}$ a quasi-martingale if there exist two stochastic processes $\{Y_t\}_{t\in\calT}$ and $\{Z_t\}_{t\in\calT}$ such that \\
\hspace*{.25cm} i) both $\{Y_t\}_{t\in\calT}$ and $\{Z_t\}_{t\in\calT}$ are adapted to $\{\calF_t\}_{t\in\calT}$, \\
\hspace*{.25cm} ii) $\{Y_t\}_{t\in\calT}$ is a martingale and $\{Z_t\}_{t\in\calT}$ has bounded variations on $\calT$ a.s.,\\
\hspace*{.25cm} iii) $X_t = Y_t + Z_t$, for all $t\in\calT$ a.s.. 
\end{definition}

\begin{lemma}[The Quasi-martingale Convergence Theorem; {\cite[Theorem 9.4 \& Proposition 9.5]{Met_82}}] \label{lem:quasi_conv}
Let $(u_t)_{t\ge 1}$ be a nonnegative discrete-time stochastic process on a probability space $(\Omega,\calA,\bbP)$, i.e., $u_t\ge 0 \;a.s.$, for all $t\ge 1$. Let $\{\calF_t\}_{t\ge 1}$ be a filtration to which $(u_t)_{t\ge 1}$ is adapted. Define another binary stochastic process $(\delta_t)_{t\ge 1}$ as
\begin{equation}
\delta_t = \begin{cases}
1, & \mbox{if} \;\; \bbE[u_{t+1}-u_t|\calF_t] >0 \\
0, & \mbox{otherwise}
\end{cases}.
\end{equation}
If $\sum_{t=1}^\infty \bbE[\delta_t(u_{t+1}-u_t)]<\infty$, then $u_t\convas u$, where $u$ is integrable on $(\Omega,\calF,\bbP)$ and nonnegative a.s.. Furthermore, $(u_t)_{t\ge 1}$ is a quasi-martingale and
\begin{equation}
\sum_{t=1}^\infty \bbE[\abs{\bbE[u_{t+1}-u_t|\calF_t]}] < \infty.
\end{equation}
\end{lemma} 

\begin{lemma}[Donsker's Theorem; {\cite[Section 19.2]{Vaart_00}}]\label{lem:Donsker}
Let $X_1,\ldots,X_n$ be \iid generated from a distribution $\bbP$. Define the empirical distribution $\bbP_n \defeq \frac{1}{n}\sum_{i=1}^n\delta_{X_i}$. For a measurable function $f$, define $\bbP_n f$ and $\bbP f$ as the expectations of $f$ under the distributions $\bbP_n$ and $\bbP$ respectively. Define an empirical process $G_n(f) \defeq \sqrt{n}(\bbP_n f - \bbP f)$, $f\in\calF$, where $\calF$ is a class of measurable functions. $\calF$ is $\bbP$-Donsker if and only if the sequence of empirical processes $\{G_n\}_{n\ge 1}$ converges in distribution to a zero-mean Gaussian process $G$ tight in $\ell^\infty(\calF)$, where $\ell^\infty(\calF)$ is the space of all real-valued and bounded functionals defined on $\calF$ equipped with the uniform norm on $\calF$, denoted as $\norm{\cdot}_\calF$. 
%\begin{equation}
%\ell^\infty(\calF) = \left\{F:\calF\to\bbR\,\bigg|\,\norm{F}_\calF <\infty\right\}.
%\end{equation}
%where $\norm{\cdot}_\calF$ denotes the uniform norm on $\calF$. 
Moreover, in such case, we have $\bbE\norm{G_n}_\calF\to\bbE\norm{G}_\calF$. 
\end{lemma}

\begin{lemma}[Glivenko-Cantelli theorem; {\cite[Section 19.2]{Vaart_00}}]\label{lem:GlivenkoCantelli}
Let $\bbP_n$ and $\bbP$ be the distributions defined as in Lemma~\ref{lem:Donsker}. A class of measurable functions $\calF$ is $\bbP$-Glivenko-Cantelli if and only if $\;\sup_{f\in\calF}\abs{\bbP_n f - \bbP f}\convas 0$. %where $\norm{\cdot}_\calF$ is the uniform norm on $\calF$.
\end{lemma}

\begin{lemma}[A sufficient condition for $\bbP$-Glivenko-Cantelli and $\bbP$-Donsker classes; {\cite[Example 19.7]{Vaart_00}}]\label{lem:suff_Donsker}
Define a probability space $(\calX,\calA,\bbP)$. Let $\calF = \{f_\theta:\calX\to\bbR\,|\,\theta\in\Theta\}$ be a class of measurable functions, where %$\calX\subseteq\bbR^n$ and 
$\Theta$ is a bounded subset in $\bbR^d$. If there exists a universal constant $K>0$ such that %measurable function $m:\calX\to\bbR$ such that 
\begin{equation}
\abs{f_{\theta_1}(x) - f_{\theta_2}(x)} \le K\norm{\theta_1 - \theta_2}, \;\forall\,\theta_1,\theta_2\in\Theta, \;\forall\,x\in\calX,
\end{equation}
where $\norm{\cdot}$ is a general vector norm in $\bbR^d$, then $\calF$ is both $\bbP$-Glivenko-Cantelli and $\bbP$-Donsker. 
\end{lemma}

\begin{lemma}[{\cite[Lemma 8]{Mairal_10}}]\label{lem:conv_nonneg}
Let $(a_n)$, $(b_n)$ be two nonnegative sequences. Suppose $\sum_{n=1}^\infty a_n = \infty$ and $\sum_{n=1}^\infty a_nb_n <\infty$, and $\exists\,N\in\bbN$ and $K>0$ such that for all $n\ge N$, $|b_{n+1}-b_n|\le Ka_n$. Then $(b_n)$ converges and $\lim_{n\to\infty} b_n = 0$.
\end{lemma}
%\begin{proof}
%Suppose $\lim_{n\to\infty} b_n$ does not exist, then $\limsup b_n > \liminf b_n \ge 0$. Thus $\exists\,\epsilon>0$ such that $\limsup b_n \ge \epsilon$. Consider two sequences of indices $(m_j)$ and $(k_j)$ such that i) $m_j < k_j < m_{j+1}$ and ii) $b_n > \epsilon/3$ if $m_j \le n < k_j$ and $b_n \le \epsilon/3$ if $k_j \le n < m_{j+1}$, $\forall\,j\in\bbN$. Define $\calJ\defeq\{j\in\bbN\,|\,m_j \ge N \,\mbox{and}\, \sum_{n=m_j}^\infty a_nb_n \le \epsilon^2/(9K)\}$, then for all $j\in\calJ$ and $m_j \le n < k_j$,
%\begin{align*}
%|b_{k_j} - b_n| &\le \sum_{i=n}^{k_j-1} |b_{i+1} - b_i|
%\le K \sum_{i=n}^{k_j-1} a_i b_{i}/b_{i}
%< \frac{3K}{\epsilon} \sum_{i=n}^{k_j-1} a_ib_i 
%\le \frac{3K}{\epsilon} \cdot \frac{\epsilon^2}{9K}
%= \frac{\epsilon}{3}. 
%\end{align*}
%Thus, $b_n \le b_{k_j} + \epsilon/3 \le 2\epsilon/3$. We conclude that for a fixed $j\in\calJ$ and $n\ge m_j$, $b_n\le 2\epsilon/3$ and consequently, $\limsup b_n \le 2\epsilon/3$. This contradicts the assumption that  $\limsup b_n \ge \epsilon$. Thus, $\lim_{n\to\infty} b_n$ exists. By contradiction, it is easy to show $\lim_{n\to\infty} b_n=0$. 
%\end{proof}


\section{Additional experiment results}\label{sec:exp_res}
This section consists of two parts. In the first part, we show the convergence speeds of all the online and batch algorithms on the (contaminated) CBCL face dataset for different values of the mini-batch size $\tau$, the latent dimension $K$, the penalty parameter $\rho$ in the ADMM-based algorithms, the step-size parameter $\kappa$ in the PGD-based algorithms and the (salt and pepper) noise density parameters $\nu$ and $\tnu$ in Figure~\ref{fig:canon2} to \ref{fig:tnu2}. As mentioned in Section~\ref{sec:conv_spd}, all the convergence results on the CBCL face dataset agree with those on the synthetic dataset. In the second part, we show the quality of the denoised images of all the algorithms on the CBCL face dataset for $K=25$ and $K=100$ in Table~\ref{tab:PSNRs} (a) and (b) respectively. The corresponding running times of all the algorithms for $K=25$ and $K=100$ are shown in Table~\ref{tab:times} (a) and (b) respectively. As stated in Section~\ref{sec:img_denois}, the results when $K=25$ and $K=100$ are similar to those when $K=49$. 

\begin{figure}[p!]\centering
\subfloat[]{\includegraphics[width=.4\columnwidth,height=.32\columnwidth]{figures/CBCL_canon_batch.eps}}\hspace{1cm}
\subfloat[]{\includegraphics[width=.4\columnwidth,height=.31\columnwidth]{figures/CBCL_canon_OL.eps}}
\caption{The objective values (as a function of time) of (a) our online algorithms and their batch counterparts (b) our online algorithms and other online algorithms on the CBCL face dataset. The parameters are set according to the canonical setting.}\label{fig:canon2}
\end{figure}

\begin{figure}[p!]\centering
\subfloat[OPGD]{\includegraphics[width=.4\columnwidth,height=.32\columnwidth]{figures/CBCL_OPGD_tau.eps}}\hspace{1cm}
\subfloat[OADMM]{\includegraphics[width=.4\columnwidth,height=.32\columnwidth]{figures/CBCL_OADMM_tau.eps}}
\caption{The objective values (as a function of time) of (a) OPGD and (b) OADMM for different values of $\tau$ on the CBCL face dataset. All the other parameters are set according to the canonical setting.}\label{fig:tau2}
\end{figure}


\begin{figure}[p!]\centering
\subfloat[ $K=25$ ]{\includegraphics[width=.4\columnwidth,height=.32\columnwidth]{figures/CBCL_K25_batch.eps}}\hspace{1cm}
\subfloat[$K=25$ ]{\includegraphics[width=.4\columnwidth,height=.31\columnwidth]{figures/CBCL_K25_OL.eps}}\\
\subfloat[$K=100$ ]{\includegraphics[width=.4\columnwidth,height=.32\columnwidth]{figures/CBCL_K100_batch.eps}}\hspace{1cm}
\subfloat[$K=100$ ]{\includegraphics[width=.4\columnwidth,height=.32\columnwidth]{figures/CBCL_K100_OL.eps}}
\caption{The objective values (as a function of time) of all the algorithms for different values of $K$ on the CBCL face dataset. In (a) and (b), $K=25$. In (c) and (d), $K=100$. All the other parameters are set according to the canonical setting.}\label{fig:K2}
\end{figure}



\begin{figure}[p!]\centering
\subfloat[BADMM]{\includegraphics[width=.4\columnwidth,height=.31\columnwidth]{figures/CBCL_batch_rho.eps}}\hspace{1cm}
\subfloat[OADMM]{\includegraphics[width=.4\columnwidth,height=.32\columnwidth]{figures/CBCL_OL_rho.eps}}
\caption{The objective values (as a function of time) of (a) BADMM and (b) OADMM for different values of $\rho$ on the CBCL face dataset. All the other parameters are set according to the canonical setting.}\label{fig:rho2}
\end{figure}
 
\begin{figure}[p!]\centering
\subfloat[BPGD]{\includegraphics[width=.4\columnwidth,height=.32\columnwidth]{figures/CBCL_batch_eta.eps}}\hspace{1cm}
\subfloat[OPGD]{\includegraphics[width=.4\columnwidth,height=.32\columnwidth]{figures/CBCL_OL_eta.eps}}
\caption{The objective values (as a function of time) of (a) BPGD and (b) OPGD for different values of $\kappa$ on the CBCL face dataset. All the other parameters are set according to the canonical setting.}\label{fig:kappa2}
\end{figure}

\begin{figure}[p!]\centering
\subfloat[$\nu = 0.8$, $\tnu = 0.2$]{\includegraphics[width=.4\columnwidth,height=.32\columnwidth]{figures/CBCL_tnu_batch.eps}}\hspace{1cm}
\subfloat[$\nu = 0.9$, $\tnu = 0.3$]{\includegraphics[width=.4\columnwidth,height=.32\columnwidth]{figures/CBCL_tnu_batch2.eps}}
\caption{The objective values (as a function of time) of our online and batch algorithms on the CBCL face dataset with a larger proportion of outliers.}\label{fig:tnu2}
\end{figure}

\begin{table}[p!]\centering
\caption{PSNRs (in dB) of all the algorithms on the CBCL face dataset with different noise density.}\label{tab:PSNRs}
\begin{subfloat}[$K=25$]{
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|}
\hline
\quad&Setting 1 & Setting 2 & Setting 3\\\hline
OADMM &  11.39 $\pm$ 0.16 &  11.37 $\pm$ 0.12 &  11.35 $\pm$ 0.18\\ \hline
OPGD &  11.49 $\pm$ 0.05 &  11.43 $\pm$ 0.09 &  11.38 $\pm$ 0.06\\ \hline
BADMM &  11.51 $\pm$ 0.19 &  11.46 $\pm$ 0.07 &  11.41 $\pm$ 0.15\\ \hline
BPGD &  11.54 $\pm$ 0.07 &  11.46 $\pm$ 0.17 &  11.42 $\pm$ 0.15\\ \hline
ONMF &   5.99 $\pm$ 0.04 &   5.97 $\pm$ 0.12 &   5.97 $\pm$ 0.08\\ \hline
ORPCA &  11.26 $\pm$ 0.05 &  11.24 $\pm$ 0.11 &  11.22 $\pm$ 0.11\\ \hline
\end{tabular}}
\end{subfloat}\hspace{1cm}
\begin{subfloat}[$K=100$]{
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|}
\hline
\quad&Setting 1 & Setting 2 & Setting 3\\\hline
OADMM &  11.39 $\pm$ 0.02 &  11.35 $\pm$ 0.11 &  11.35 $\pm$ 0.06\\ \hline
OPGD &  11.51 $\pm$ 0.01 &  11.45 $\pm$ 0.09 &  11.44 $\pm$ 0.11\\ \hline
BADMM &  11.51 $\pm$ 0.11 &  11.47 $\pm$ 0.00 &  11.45 $\pm$ 0.03\\ \hline
BPGD &  11.52 $\pm$ 0.16 &  11.46 $\pm$ 0.07 &  11.45 $\pm$ 0.12\\ \hline
ONMF &   5.99 $\pm$ 0.19 &   5.97 $\pm$ 0.03 &   5.95 $\pm$ 0.05\\ \hline
ORPCA &  11.26 $\pm$ 0.03 &  11.24 $\pm$ 0.16 &  11.20 $\pm$ 0.13\\ \hline
\end{tabular}
}
\end{subfloat}
\end{table}

\begin{table}[p!]\centering
\caption{Running times (in seconds) of all the algorithms on the CBCL face dataset with different noise density.}\label{tab:times}
\begin{subfloat}[$K=25$]{
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|}
\hline
\quad&Setting 1 & Setting 2 & Setting 3\\\hline
OADMM & 420.58 $\pm$ 2.59 & 427.66 $\pm$ 4.80 & 430.02 $\pm$ 2.93\\ \hline
OPGD & 431.66 $\pm$ 2.49 & 455.15 $\pm$ 1.70 & 463.67 $\pm$ 1.12\\ \hline
BADMM & 1009.45 $\pm$ 11.27 & 1184.29 $\pm$ 10.49 & 1240.91 $\pm$ 8.21\\ \hline
BPGD & 1125.58 $\pm$ 12.83 & 1185.64 $\pm$ 13.36 & 1279.07 $\pm$ 9.08\\ \hline
ONMF & 2384.70 $\pm$ 9.59 & 2588.29 $\pm$ 14.39 & 2698.57 $\pm$ 10.24\\ \hline
ORPCA & 365.98 $\pm$ 5.29 & 382.49 $\pm$ 4.20 & 393.10 $\pm$ 4.27\\ \hline
\end{tabular}}
\end{subfloat}\hspace{1cm}
\begin{subfloat}[$K=100$]{
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|}
\hline
\quad&Setting 1 & Setting 2 & Setting 3\\\hline
OADMM & 422.97 $\pm$ 2.38 & 424.67 $\pm$ 2.65 & 434.17 $\pm$ 4.67\\ \hline
OPGD & 430.19 $\pm$ 2.27 & 448.72 $\pm$ 3.90 & 454.30 $\pm$ 4.65\\ \hline
BADMM & 1009.04 $\pm$ 8.53 & 1187.58 $\pm$ 5.06 & 1250.53 $\pm$ 4.67\\ \hline
BPGD & 1131.89 $\pm$ 7.04 & 1192.46 $\pm$ 7.43 & 1280.55 $\pm$ 7.93\\ \hline
ONMF & 2379.86 $\pm$ 15.18 & 2591.09 $\pm$ 11.91 & 2693.08 $\pm$ 12.48\\ \hline
ORPCA & 363.37 $\pm$ 3.01 & 390.58 $\pm$ 5.31 & 401.21 $\pm$ 3.27\\ \hline
\end{tabular}
}
\end{subfloat}
\end{table}


\end{document}

preamble.tex
\usepackage[mathscr]{eucal}
%\usepackage[cmex10]{amsmath}
\usepackage{epsfig,epsf,psfrag}
\usepackage{amssymb,amsmath,amsfonts,latexsym}
\usepackage{graphicx}
%\usepackage{epstopdf}
\usepackage{epstopdf}
\usepackage{bm,xcolor,url}
\usepackage[caption=false]{subfig} 
\usepackage{fixltx2e}%ordering of single and double column floats
\usepackage{array}%array and tabular environments
\usepackage{verbatim}
\usepackage{bm}
\usepackage{algpseudocode, cite}
\usepackage{algorithm}
\usepackage{verbatim}
\usepackage{textcomp}
\usepackage{mathrsfs}
%\usepackage{ntheorem}
%\usepackage{hyperref}
%\usepackage{epstopdf}
%\usepackage{setspace}
\usepackage{relsize}
%\usepackage{cleveref} 
\usepackage{subfig}
\usepackage{amsthm}
\usepackage{enumerate}
%\usepackage{subfigure}
%\usepackage{subcaption}
%\usepackage{url}
\usepackage{setspace}
\usepackage{cases}
\usepackage{footnote}
\usepackage{tabularx,tabu}
\usepackage{tablefootnote}
\usepackage{threeparttable}
\usepackage{tikz}
\usetikzlibrary{datavisualization}
\usetikzlibrary{datavisualization.formats.functions}
\usetikzlibrary{decorations.pathreplacing}
\usepackage{textcomp}
\usepackage{multicol}
\usepackage{lipsum}
%\usepackage{bold-extra}
%\usepackage{beramono}
%\usepackage{lmodern}
%\newcommand{\span}{\mathrm{span}}
 
%% To produce a tilde in url
\catcode`~=11 \def\UrlSpecials{\do\~{\kern -.15em\lower .7ex\hbox{~}\kern .04em}} \catcode`~=13 

\allowdisplaybreaks[3]

\newcommand{\calHN}{\calH\calN} 
\newcommand{\iidsim}{\widesim{iid}}
\newcommand{\vecz}{\mathbf{0}}
\newcommand{\vecone}{\mathbf{1}}
\newcommand{\Iff}{if and only if\;\,} 
\newcommand{\norm}[1]{\left\Vert#1\right\Vert}
\newcommand{\normt}[1]{\Vert#1\Vert}
\newcommand{\abst}[1]{\left\vert#1\right\vert}
\newcommand{\abs}[1]{\left\lvert#1\right\rvert}
\newcommand{\lrpar}[1]{\left(#1\right)}
\newcommand{\lrang}[1]{\left\langle#1\right\rangle}
\newcommand{\lrbrace}[1]{\left\{#1\right\}}
\newcommand{\as}{{\rm \;\;a.s.}}
\newcommand{\nn}{\nonumber} 
\newcommand{\cequal}{\stackrel{\mathrm{c}}{=}}
\newcommand{\dequal}{\stackrel{\mathrm{d}}{=}}
\newcommand{\St}{\mathrm{subject\; to}}
%\newcommand{\defeq}{\stackrel{\scriptscriptstyle\Delta}{=}}
\newcommand{\defeq}{\triangleq}
\newcommand{\eqcst}{\stackrel{{\rm c}}{=}}
\newcommand\numberthis{\addtocounter{equation}{1}\tag{\theequation}} 
\newcommand{\prox}{\mathbf{prox}}
\newcommand{\dom}{\mathbf{dom}}
\newcommand{\wrt}{w.r.t.\,}
\newcommand{\tilell}{\widetilde{\ell}}
\newcommand{\widesim}[2][1.5]{\mathrel{\overset{#2}{\scalebox{#1}[1]{$\sim$}}}}
\newcommand{\convu}{\xrightarrow{{\rm u}}}
\newcommand{\inter}{\mathbf{int}\,}
\newcommand{\bdr}{\mathbf{bd}\,}
\newcommand{\cl}{\mathbf{cl}\,}
\newcommand{\conv}{\mathbf{conv}\,}
\newcommand{\convcl}{\overline{\mathbf{conv}\,}}
\newcommand{\convinf}[1]{\xrightarrow{#1\to\infty}}
\newcommand{\diam}{\mathbf{diam}\,}
\newcommand{\dist}{\mathbf{dist}\,}
\newcommand{\hpartial}{\hat{\partial}}
\newcommand{\scHN}{\mathscr{H\hspace{-.15cm}N\hspace{-0cm}}}
\newcommand{\snr}{\mathtt{SNR}}
%\newcommand{\scSt}{\mathscr{St}}
\newcommand{\Section}[1]{\vspace{-8pt}\section{\hskip 0em~~#1}\vspace{-1pt}}
\newcommand{\Subsection}[1]{\vspace{-8pt}\subsection{\hskip -1em~~#1}\vspace{-3pt}}
\newcommand{\eps}{\epsilon}
\font\btt=rm-lmtk10

%Bold bar
\newcommand{\barba}{\overline{\bf{a}}}
\newcommand{\barbb}{\overline{\bf b}}
\newcommand{\barbc}{\overline{\bf c}}
\newcommand{\barbd}{\overline{\bf d}}
\newcommand{\barbe}{\overline{\bf e}}
\newcommand{\barbf}{\overline{\bf f}}
\newcommand{\barbg}{\overline{\bf g}}
\newcommand{\barbh}{\overline{\bf h}}
\newcommand{\barbi}{\overline{\bf i}}
\newcommand{\barbj}{\overline{\bf j}}
\newcommand{\barbk}{\overline{\bf k}}
\newcommand{\barbl}{\overline{\bf l}}
\newcommand{\barbm}{\overline{\bf m}}
\newcommand{\barbn}{\overline{\bf n}}
\newcommand{\barbo}{\overline{\bf o}}
\newcommand{\barbp}{\overline{\bf p}}
\newcommand{\barbq}{\overline{\bf q}}
\newcommand{\barbr}{\overline{\bf r}}
\newcommand{\barbs}{\overline{\bf s}}
\newcommand{\barbt}{\overline{\bf t}}
\newcommand{\barbu}{\overline{\bf u}}
\newcommand{\barbv}{\overline{\bf v}}
\newcommand{\barbw}{\overline{\bf w}}
\newcommand{\barbx}{\overline{\bf x}}
\newcommand{\barby}{\overline{\bf y}}
\newcommand{\barbz}{\overline{\bf z}}

\newcommand{\barbA}{\overline{\bf A}}
\newcommand{\barbB}{\overline{\bf B}}
\newcommand{\barbC}{\overline{\bf C}}
\newcommand{\barbD}{\overline{\bf D}}
\newcommand{\barbE}{\overline{\bf E}}
\newcommand{\barbF}{\overline{\bf F}}
\newcommand{\barbG}{\overline{\bf G}}
\newcommand{\barbH}{\overline{\bf H}}
\newcommand{\barbI}{\overline{\bf I}}
\newcommand{\barbJ}{\overline{\bf J}}
\newcommand{\barbK}{\overline{\bf K}}
\newcommand{\barbL}{\overline{\bf L}}
\newcommand{\barbM}{\overline{\bf M}}
\newcommand{\barbN}{\overline{\bf N}}
\newcommand{\barbO}{\overline{\bf O}}
\newcommand{\barbP}{\overline{\bf P}}
\newcommand{\barbQ}{\overline{\bf Q}}
\newcommand{\barbR}{\overline{\bf R}}
\newcommand{\barbS}{\overline{\bf S}}
\newcommand{\barbT}{\overline{\bf T}}
\newcommand{\barbU}{\overline{\bf U}}
\newcommand{\barbV}{\overline{\bf V}}
\newcommand{\barbW}{\overline{\bf W}}
\newcommand{\barbX}{\overline{\bf X}}
\newcommand{\barbY}{\overline{\bf Y}}
\newcommand{\barbZ}{\overline{\bf Z}}

\newcommand{\barell}{\overline{\ell}}
\newcommand{\barkappa}{\overline{\kappa}}

% Calligraphic stuff
\newcommand{\calA}{\mathcal{A}}
\newcommand{\calB}{\mathcal{B}}
\newcommand{\calC}{\mathcal{C}}
\newcommand{\calD}{\mathcal{D}}
\newcommand{\calE}{\mathcal{E}}
\newcommand{\calF}{\mathcal{F}}
\newcommand{\calG}{\mathcal{G}}
\newcommand{\calH}{\mathcal{H}}
\newcommand{\calI}{\mathcal{I}}
\newcommand{\calJ}{\mathcal{J}}
\newcommand{\calK}{\mathcal{K}}
\newcommand{\calL}{\mathcal{L}}
\newcommand{\calM}{\mathcal{M}}
\newcommand{\calN}{\mathcal{N}}
\newcommand{\calO}{\mathcal{O}}
\newcommand{\calP}{\mathcal{P}}
\newcommand{\calQ}{\mathcal{Q}}
\newcommand{\calR}{\mathcal{R}}
\newcommand{\calS}{\mathcal{S}}
\newcommand{\calT}{\mathcal{T}}
\newcommand{\calU}{\mathcal{U}}
\newcommand{\calV}{\mathcal{V}}
\newcommand{\calW}{\mathcal{W}}
\newcommand{\calX}{\mathcal{X}}
\newcommand{\calY}{\mathcal{Y}}
\newcommand{\calZ}{\mathcal{Z}}

\newcommand{\hatcalA}{\widehat{\calA}} 
\newcommand{\hatcalB}{\widehat{\calB}} 
\newcommand{\hatcalC}{\widehat{\calC}} 
\newcommand{\hatcalD}{\widehat{\calD}} 
\newcommand{\hatcalE}{\widehat{\calE}} 
\newcommand{\hatcalF}{\widehat{\calF}} 
\newcommand{\hatcalG}{\widehat{\calG}} 
\newcommand{\hatcalH}{\widehat{\calH}} 
\newcommand{\hatcalI}{\widehat{\calI}} 
\newcommand{\hatcalJ}{\widehat{\calJ}} 
\newcommand{\hatcalK}{\widehat{\calK}} 
\newcommand{\hatcalL}{\widehat{\calL}} 
\newcommand{\hatcalM}{\widehat{\calM}} 
\newcommand{\hatcalN}{\widehat{\calN}} 
\newcommand{\hatcalO}{\widehat{\calO}} 
\newcommand{\hatcalP}{\widehat{\calP}} 
\newcommand{\hatcalQ}{\widehat{\calQ}} 
\newcommand{\hatcalR}{\widehat{\calR}} 
\newcommand{\hatcalS}{\widehat{\calS}} 
\newcommand{\hatcalT}{\widehat{\calT}} 
\newcommand{\hatcalU}{\widehat{\calU}} 
\newcommand{\hatcalV}{\widehat{\calV}} 
\newcommand{\hatcalW}{\widehat{\calW}} 
\newcommand{\hatcalX}{\widehat{\calX}} 
\newcommand{\hatcalY}{\widehat{\calY}} 
\newcommand{\hatcalZ}{\widehat{\calZ}} 

\newcommand{\tilcalA}{\widetilde{\calA}} 
\newcommand{\tilcalB}{\widetilde{\calB}} 
\newcommand{\tilcalC}{\widetilde{\calC}} 
\newcommand{\tilcalD}{\widetilde{\calD}} 
\newcommand{\tilcalE}{\widetilde{\calE}} 
\newcommand{\tilcalF}{\widetilde{\calF}} 
\newcommand{\tilcalG}{\widetilde{\calG}} 
\newcommand{\tilcalH}{\widetilde{\calH}} 
\newcommand{\tilcalI}{\widetilde{\calI}} 
\newcommand{\tilcalJ}{\widetilde{\calJ}} 
\newcommand{\tilcalK}{\widetilde{\calK}} 
\newcommand{\tilcalL}{\widetilde{\calL}} 
\newcommand{\tilcalM}{\widetilde{\calM}} 
\newcommand{\tilcalN}{\widetilde{\calN}} 
\newcommand{\tilcalO}{\widetilde{\calO}} 
\newcommand{\tilcalP}{\widetilde{\calP}} 
\newcommand{\tilcalQ}{\widetilde{\calQ}} 
\newcommand{\tilcalR}{\widetilde{\calR}} 
\newcommand{\tilcalS}{\widetilde{\calS}} 
\newcommand{\tilcalT}{\widetilde{\calT}} 
\newcommand{\tilcalU}{\widetilde{\calU}} 
\newcommand{\tilcalV}{\widetilde{\calV}} 
\newcommand{\tilcalW}{\widetilde{\calW}} 
\newcommand{\tilcalX}{\widetilde{\calX}} 
\newcommand{\tilcalY}{\widetilde{\calY}} 
\newcommand{\tilcalZ}{\widetilde{\calZ}} 

\newcommand{\barcalA}{\overline{\calA}} 
\newcommand{\barcalB}{\overline{\calB}} 
\newcommand{\barcalC}{\overline{\calC}} 
\newcommand{\barcalD}{\overline{\calD}} 
\newcommand{\barcalE}{\overline{\calE}} 
\newcommand{\barcalF}{\overline{\calF}} 
\newcommand{\barcalG}{\overline{\calG}} 
\newcommand{\barcalH}{\overline{\calH}} 
\newcommand{\barcalI}{\overline{\calI}} 
\newcommand{\barcalJ}{\overline{\calJ}} 
\newcommand{\barcalK}{\overline{\calK}} 
\newcommand{\barcalL}{\overline{\calL}} 
\newcommand{\barcalM}{\overline{\calM}} 
\newcommand{\barcalN}{\overline{\calN}} 
\newcommand{\barcalO}{\overline{\calO}} 
\newcommand{\barcalP}{\overline{\calP}} 
\newcommand{\barcalQ}{\overline{\calQ}} 
\newcommand{\barcalR}{\overline{\calR}} 
\newcommand{\barcalS}{\overline{\calS}} 
\newcommand{\barcalT}{\overline{\calT}} 
\newcommand{\barcalU}{\overline{\calU}} 
\newcommand{\barcalV}{\overline{\calV}} 
\newcommand{\barcalW}{\overline{\calW}} 
\newcommand{\barcalX}{\overline{\calX}} 
\newcommand{\barcalY}{\overline{\calY}} 
\newcommand{\barcalZ}{\overline{\calZ}} 

% Boldface stuff
\newcommand{\ba}{\mathbf{a}}
\newcommand{\bA}{\mathbf{A}}
\newcommand{\bb}{\mathbf{b}}
\newcommand{\bB}{\mathbf{B}}
\newcommand{\bc}{\mathbf{c}}
\newcommand{\bC}{\mathbf{C}}
\newcommand{\bd}{\mathbf{d}}
\newcommand{\bD}{\mathbf{D}}
\newcommand{\be}{\mathbf{e}}
\newcommand{\bE}{\mathbf{E}}
\newcommand{\boldf}{\mathbf{f}}
\newcommand{\bF}{\mathbf{F}}
\newcommand{\bg}{\mathbf{g}}
\newcommand{\bG}{\mathbf{G}}
\newcommand{\bh}{\mathbf{h}}
\newcommand{\bH}{\mathbf{H}}
\newcommand{\bi}{\mathbf{i}}
\newcommand{\bI}{\mathbf{I}}
\newcommand{\bj}{\mathbf{j}}
\newcommand{\bJ}{\mathbf{J}}
\newcommand{\bk}{\mathbf{k}}
\newcommand{\bK}{\mathbf{K}}
\newcommand{\bl}{\mathbf{l}}
\newcommand{\bL}{\mathbf{L}}
\newcommand{\boldm}{\mathbf{m}}
\newcommand{\bM}{\mathbf{M}}
\newcommand{\bn}{\mathbf{n}}
\newcommand{\bN}{\mathbf{N}}
\newcommand{\bo}{\mathbf{o}}
\newcommand{\bO}{\mathbf{O}}
\newcommand{\bp}{\mathbf{p}}
\newcommand{\bP}{\mathbf{P}}
\newcommand{\bq}{\mathbf{q}}
\newcommand{\bQ}{\mathbf{Q}}
\newcommand{\br}{\mathbf{r}}
\newcommand{\bR}{\mathbf{R}}
\newcommand{\bs}{\mathbf{s}}
\newcommand{\bS}{\mathbf{S}}
\newcommand{\bt}{\mathbf{t}}
\newcommand{\bT}{\mathbf{T}}
\newcommand{\bu}{\mathbf{u}}
\newcommand{\bU}{\mathbf{U}}
\newcommand{\bv}{\mathbf{v}}
\newcommand{\bV}{\mathbf{V}}
\newcommand{\bw}{\mathbf{w}}
\newcommand{\bW}{\mathbf{W}}
\newcommand{\bx}{\mathbf{x}}
\newcommand{\bX}{\mathbf{X}}
\newcommand{\by}{\mathbf{y}}
\newcommand{\bY}{\mathbf{Y}}
\newcommand{\bz}{\mathbf{z}}
\newcommand{\bZ}{\mathbf{Z}}

% Roman stuff
\newcommand{\rma}{\mathrm{a}}
\newcommand{\rmA}{\mathrm{A}}
\newcommand{\rmb}{\mathrm{b}}
\newcommand{\rmB}{\mathrm{B}}
\newcommand{\rmc}{\mathrm{c}}
\newcommand{\rmC}{\mathrm{C}}
\newcommand{\rmd}{\mathrm{d}}
\newcommand{\rmD}{\mathrm{D}}
\newcommand{\rme}{\mathrm{e}}
\newcommand{\rmE}{\mathrm{E}}
\newcommand{\rmf}{\mathrm{f}}
\newcommand{\rmF}{\mathrm{F}}
\newcommand{\rmg}{\mathrm{g}}
\newcommand{\rmG}{\mathrm{G}}
\newcommand{\rmh}{\mathrm{h}}
\newcommand{\rmH}{\mathrm{H}}
\newcommand{\rmi}{\mathrm{i}}
\newcommand{\rmI}{\mathrm{I}}
\newcommand{\rmj}{\mathrm{j}}
\newcommand{\rmJ}{\mathrm{J}}
\newcommand{\rmk}{\mathrm{k}}
\newcommand{\rmK}{\mathrm{K}}
\newcommand{\rml}{\mathrm{l}}
\newcommand{\rmL}{\mathrm{L}}
\newcommand{\rmm}{\mathrm{m}}
\newcommand{\rmM}{\mathrm{M}}
\newcommand{\rmn}{\mathrm{n}}
\newcommand{\rmN}{\mathrm{N}}
\newcommand{\rmo}{\mathrm{o}}
\newcommand{\rmO}{\mathrm{O}}
\newcommand{\rmp}{\mathrm{p}}
\newcommand{\rmP}{\mathrm{P}}
\newcommand{\rmq}{\mathrm{q}}
\newcommand{\rmQ}{\mathrm{Q}}
\newcommand{\rmr}{\mathrm{r}}
\newcommand{\rmR}{\mathrm{R}}
\newcommand{\rms}{\mathrm{s}}
\newcommand{\rmS}{\mathrm{S}}
\newcommand{\rmt}{\mathrm{t}}
\newcommand{\rmT}{\mathrm{T}}
\newcommand{\rmu}{\mathrm{u}}
\newcommand{\rmU}{\mathrm{U}}
\newcommand{\rmv}{\mathrm{v}}
\newcommand{\rmV}{\mathrm{V}}
\newcommand{\rmw}{\mathrm{w}}
\newcommand{\rmW}{\mathrm{W}}
\newcommand{\rmx}{\mathrm{x}}
\newcommand{\rmX}{\mathrm{X}}
\newcommand{\rmy}{\mathrm{y}}
\newcommand{\rmY}{\mathrm{Y}}
\newcommand{\rmz}{\mathrm{z}}
\newcommand{\rmZ}{\mathrm{Z}}


% Numbers bb font
\newcommand{\bbA}{\mathbb{A}}
\newcommand{\bbB}{\mathbb{B}}
\newcommand{\bbC}{\mathbb{C}}
\newcommand{\bbD}{\mathbb{D}}
\newcommand{\bbE}{\mathbb{E}}
\newcommand{\bbF}{\mathbb{F}}
\newcommand{\bbG}{\mathbb{G}}
\newcommand{\bbH}{\mathbb{H}}
\newcommand{\bbI}{\mathbb{I}}
\newcommand{\bbJ}{\mathbb{J}}
\newcommand{\bbK}{\mathbb{K}}
\newcommand{\bbL}{\mathbb{L}}
\newcommand{\bbM}{\mathbb{M}}
\newcommand{\bbN}{\mathbb{N}}
\newcommand{\bbO}{\mathbb{O}}
\newcommand{\bbP}{\mathbb{P}}
\newcommand{\bbQ}{\mathbb{Q}}
\newcommand{\bbR}{\mathbb{R}}
\newcommand{\bbS}{\mathbb{S}}
\newcommand{\bbT}{\mathbb{T}}
\newcommand{\bbU}{\mathbb{U}}
\newcommand{\bbV}{\mathbb{V}}
\newcommand{\bbW}{\mathbb{W}}
\newcommand{\bbX}{\mathbb{X}}
\newcommand{\bbY}{\mathbb{Y}}
\newcommand{\bbZ}{\mathbb{Z}}


% Mathfrak font
\newcommand{\frakA}{\mathfrak{A}}
\newcommand{\frakB}{\mathfrak{B}}
\newcommand{\frakC}{\mathfrak{C}}
\newcommand{\frakD}{\mathfrak{D}}
\newcommand{\frakE}{\mathfrak{E}}
\newcommand{\frakF}{\mathfrak{F}}
\newcommand{\frakG}{\mathfrak{G}}
\newcommand{\frakH}{\mathfrak{H}}
\newcommand{\frakI}{\mathfrak{I}}
\newcommand{\frakJ}{\mathfrak{J}}
\newcommand{\frakK}{\mathfrak{K}}
\newcommand{\frakL}{\mathfrak{L}}
\newcommand{\frakM}{\mathfrak{M}}
\newcommand{\frakN}{\mathfrak{N}}
\newcommand{\frakO}{\mathfrak{O}}
\newcommand{\frakP}{\mathfrak{P}}
\newcommand{\frakQ}{\mathfrak{Q}}
\newcommand{\frakR}{\mathfrak{R}}
\newcommand{\frakS}{\mathfrak{S}}
\newcommand{\frakT}{\mathfrak{T}}
\newcommand{\frakU}{\mathfrak{U}}
\newcommand{\frakV}{\mathfrak{V}}
\newcommand{\frakW}{\mathfrak{W}}
\newcommand{\frakX}{\mathfrak{X}}
\newcommand{\frakY}{\mathfrak{Y}}
\newcommand{\frakZ}{\mathfrak{Z}}


% Mathscr
\newcommand{\scA}{\mathscr{A}}
\newcommand{\scB}{\mathscr{B}}
\newcommand{\scC}{\mathscr{C}}
\newcommand{\scD}{\mathscr{D}}
\newcommand{\scE}{\mathscr{E}}
\newcommand{\scF}{\mathscr{F}}
\newcommand{\scG}{\mathscr{G}}
\newcommand{\scH}{\mathscr{H}}
\newcommand{\scI}{\mathscr{I}}
\newcommand{\scJ}{\mathscr{J}}
\newcommand{\scK}{\mathscr{K}}
\newcommand{\scL}{\mathscr{L}}
\newcommand{\scM}{\mathscr{M}}
\newcommand{\scN}{\mathscr{N}}
\newcommand{\scO}{\mathscr{O}}
\newcommand{\scP}{\mathscr{P}}
\newcommand{\scQ}{\mathscr{Q}}
\newcommand{\scR}{\mathscr{R}}
\newcommand{\scS}{\mathscr{S}}
\newcommand{\scT}{\mathscr{T}}
\newcommand{\scU}{\mathscr{U}}
\newcommand{\scV}{\mathscr{V}}
\newcommand{\scW}{\mathscr{W}}
\newcommand{\scX}{\mathscr{X}}
\newcommand{\scY}{\mathscr{Y}}
\newcommand{\scZ}{\mathscr{Z}}

% San serif font
\DeclareMathAlphabet{\mathbsf}{OT1}{cmss}{bx}{n}
\DeclareMathAlphabet{\mathssf}{OT1}{cmss}{m}{sl}% slanted sans serif
\newcommand{\rva}{\mathsf{a}}
\newcommand{\rvA}{\mathsf{A}}
\newcommand{\rvba}{\mathbsf{a}}
\newcommand{\rvbA}{\mathbsf{A}}
\newcommand{\rvb}{\mathsf{b}}
\newcommand{\rvB}{\mathsf{B}}
\newcommand{\rvbb}{\mathbsf{b}}
\newcommand{\rvbB}{\mathbsf{B}}
\newcommand{\rvc}{\mathsf{c}}
\newcommand{\rvC}{\mathsf{C}}
\newcommand{\rvbc}{\mathbsf{c}}
\newcommand{\rvbC}{\mathbsf{C}}
\newcommand{\rvd}{\mathsf{d}}
\newcommand{\rvD}{\mathsf{D}}
\newcommand{\rvbd}{\mathbsf{d}}
\newcommand{\rvbD}{\mathbsf{D}}
\newcommand{\rve}{\mathsf{e}}
\newcommand{\rvE}{\mathsf{E}}
\newcommand{\rvbe}{\mathbsf{e}}
\newcommand{\rvbE}{\mathbsf{E}}
\newcommand{\rvf}{\mathsf{f}}
\newcommand{\rvF}{\mathsf{F}}
\newcommand{\rvbf}{\mathbsf{f}}
\newcommand{\rvbF}{\mathbsf{F}}
\newcommand{\rvg}{\mathsf{g}}
\newcommand{\rvG}{\mathsf{G}}
\newcommand{\rvbg}{\mathbsf{g}}
\newcommand{\rvbG}{\mathbsf{G}}
\newcommand{\rvh}{\mathsf{h}}
\newcommand{\rvH}{\mathsf{H}}
\newcommand{\rvbh}{\mathbsf{h}}
\newcommand{\rvbH}{\mathbsf{H}}
\newcommand{\rvi}{\mathsf{i}}
\newcommand{\rvI}{\mathsf{I}}
\newcommand{\rvbi}{\mathbsf{i}}
\newcommand{\rvbI}{\mathbsf{I}}
\newcommand{\rvj}{\mathsf{j}}
\newcommand{\rvJ}{\mathsf{J}}
\newcommand{\rvbj}{\mathbsf{j}}
\newcommand{\rvbJ}{\mathbsf{J}}
\newcommand{\rvk}{\mathsf{k}}
\newcommand{\rvK}{\mathsf{K}}
\newcommand{\rvbk}{\mathbsf{k}}
\newcommand{\rvbK}{\mathbsf{K}}
\newcommand{\rvl}{\mathsf{l}}
\newcommand{\rvL}{\mathsf{L}}
\newcommand{\rvbl}{\mathbsf{l}}
\newcommand{\rvbL}{\mathbsf{L}}
\newcommand{\rvm}{\mathsf{m}}
\newcommand{\rvM}{\mathsf{M}}
\newcommand{\rvbm}{\mathbsf{m}}
\newcommand{\rvbM}{\mathbsf{M}}
\newcommand{\rvn}{\mathsf{n}}
\newcommand{\rvN}{\mathsf{N}}
\newcommand{\rvbn}{\mathbsf{n}}
\newcommand{\rvbN}{\mathbsf{N}}
\newcommand{\rvo}{\mathsf{o}}
\newcommand{\rvO}{\mathsf{O}}
\newcommand{\rvbo}{\mathbsf{o}}
\newcommand{\rvbO}{\mathbsf{O}}
\newcommand{\rvp}{\mathsf{p}}
\newcommand{\rvP}{\mathsf{P}}
\newcommand{\rvbp}{\mathbsf{p}}
\newcommand{\rvbP}{\mathbsf{P}}
\newcommand{\rvq}{\mathsf{q}}
\newcommand{\rvQ}{\mathsf{Q}}
\newcommand{\rvbq}{\mathbsf{q}}
\newcommand{\rvbQ}{\mathbsf{Q}}
\newcommand{\rvr}{\mathsf{r}}
\newcommand{\rvR}{\mathsf{R}}
\newcommand{\rvbr}{\mathbsf{r}}
\newcommand{\rvbR}{\mathbsf{R}}
\newcommand{\rvs}{\mathsf{s}}
\newcommand{\rvS}{\mathsf{S}}
\newcommand{\rvbs}{\mathbsf{s}}
\newcommand{\rvbS}{\mathbsf{S}}
\newcommand{\rvt}{\mathsf{t}}
\newcommand{\rvT}{\mathsf{T}}
\newcommand{\rvbt}{\mathbsf{t}}
\newcommand{\rvbT}{\mathbsf{T}}
\newcommand{\rvu}{\mathsf{u}}
\newcommand{\rvU}{\mathsf{U}}
\newcommand{\rvbu}{\mathbsf{u}}
\newcommand{\rvbU}{\mathbsf{U}}
\newcommand{\rvv}{\mathsf{v}}
\newcommand{\rvV}{\mathsf{V}}
\newcommand{\rvbv}{\mathbsf{v}}
\newcommand{\rvbV}{\mathbsf{V}}
\newcommand{\rvw}{\mathsf{w}}
\newcommand{\rvW}{\mathsf{W}}
\newcommand{\rvbw}{\mathbsf{w}}
\newcommand{\rvbW}{\mathbsf{W}}
\newcommand{\rvx}{\mathsf{x}}
\newcommand{\rvX}{\mathsf{X}}
\newcommand{\rvbx}{\mathbsf{x}}
\newcommand{\rvbX}{\mathbsf{X}}
\newcommand{\rvy}{\mathsf{y}}
\newcommand{\rvY}{\mathsf{Y}}
\newcommand{\rvby}{\mathbsf{y}}
\newcommand{\rvbY}{\mathbsf{Y}}
\newcommand{\rvz}{\mathsf{z}}
\newcommand{\rvZ}{\mathsf{Z}}
\newcommand{\rvbz}{\mathbsf{z}}
\newcommand{\rvbZ}{\mathbsf{Z}}

\newcommand{\rvTh}{\ssfTheta}
\newcommand{\svTh}{\Theta}
\newcommand{\rvbTh}{\bsfTheta}
\newcommand{\svbTh}{\boldsymbol{\Theta}}
\newcommand{\rvPh}{\ssfPhi}
\newcommand{\svPh}{\Phi}
\newcommand{\rvbPh}{\bsfPhi}
\newcommand{\svbPh}{\boldsymbol{\Phi}}

% define some useful uppercase Greek letters in regular and bold sf
\DeclareSymbolFont{bsfletters}{OT1}{cmss}{bx}{n}  
\DeclareSymbolFont{ssfletters}{OT1}{cmss}{m}{n}
\DeclareMathSymbol{\bsfGamma}{0}{bsfletters}{'000}
\DeclareMathSymbol{\ssfGamma}{0}{ssfletters}{'000}
\DeclareMathSymbol{\bsfDelta}{0}{bsfletters}{'001}
\DeclareMathSymbol{\ssfDelta}{0}{ssfletters}{'001}
\DeclareMathSymbol{\bsfTheta}{0}{bsfletters}{'002}
\DeclareMathSymbol{\ssfTheta}{0}{ssfletters}{'002}
\DeclareMathSymbol{\bsfLambda}{0}{bsfletters}{'003}
\DeclareMathSymbol{\ssfLambda}{0}{ssfletters}{'003}
\DeclareMathSymbol{\bsfXi}{0}{bsfletters}{'004}
\DeclareMathSymbol{\ssfXi}{0}{ssfletters}{'004}
\DeclareMathSymbol{\bsfPi}{0}{bsfletters}{'005}
\DeclareMathSymbol{\ssfPi}{0}{ssfletters}{'005}
\DeclareMathSymbol{\bsfSigma}{0}{bsfletters}{'006}
\DeclareMathSymbol{\ssfSigma}{0}{ssfletters}{'006}
\DeclareMathSymbol{\bsfUpsilon}{0}{bsfletters}{'007}
\DeclareMathSymbol{\ssfUpsilon}{0}{ssfletters}{'007}
\DeclareMathSymbol{\bsfPhi}{0}{bsfletters}{'010}
\DeclareMathSymbol{\ssfPhi}{0}{ssfletters}{'010}
\DeclareMathSymbol{\bsfPsi}{0}{bsfletters}{'011}
\DeclareMathSymbol{\ssfPsi}{0}{ssfletters}{'011}
\DeclareMathSymbol{\bsfOmega}{0}{bsfletters}{'012}
\DeclareMathSymbol{\ssfOmega}{0}{ssfletters}{'012}

% Hat and Tilde
\newcommand{\hata}{\widehat{a}}
\newcommand{\hatA}{\widehat{A}}
\newcommand{\tila}{\widetilde{a}}
\newcommand{\tilA}{\widetilde{A}}
\newcommand{\hatba}{\widehat{\ba}}
\newcommand{\hatbA}{\widehat{\bA}}
\newcommand{\tilba}{\widetilde{\ba}}
\newcommand{\tilbA}{\widetilde{\bA}}

\newcommand{\hatb}{\widehat{b}}
\newcommand{\hatB}{\widehat{B}}
\newcommand{\tilb}{\widetilde{b}}
\newcommand{\tilB}{\widetilde{B}}
\newcommand{\hatbb}{\widehat{\bb}}
\newcommand{\hatbB}{\widehat{\bB}}
\newcommand{\tilbb}{\widetilde{\bb}}
\newcommand{\tilbB}{\widetilde{\bB}}

\newcommand{\hatc}{\widehat{c}}
\newcommand{\hatC}{\widehat{C}}
\newcommand{\tilc}{\widetilde{c}}
\newcommand{\tilC}{\widetilde{C}}
\newcommand{\hatbc}{\widehat{\bc}}
\newcommand{\hatbC}{\widehat{\bC}}
\newcommand{\tilbc}{\widetilde{\bc}}
\newcommand{\tilbC}{\widetilde{\bC}}

\newcommand{\hatd}{\widehat{d}}
\newcommand{\hatD}{\widehat{D}}
\newcommand{\tild}{\widetilde{d}}
\newcommand{\tilD}{\widetilde{D}}
\newcommand{\hatbd}{\widehat{\bd}}
\newcommand{\hatbD}{\widehat{\bD}}
\newcommand{\tilbd}{\widetilde{\bd}}
\newcommand{\tilbD}{\widetilde{\bD}}

\newcommand{\hate}{\widehat{e}}
\newcommand{\hatE}{\widehat{E}}
\newcommand{\tile}{\widetilde{e}}
\newcommand{\tilE}{\widetilde{E}}
\newcommand{\hatbe}{\widehat{\be}}
\newcommand{\hatbE}{\widehat{\bE}}
\newcommand{\tilbe}{\widetilde{\be}}
\newcommand{\tilbE}{\widetilde{\bE}}

\newcommand{\hatf}{\widehat{f}}
\newcommand{\hatF}{\widehat{F}}
\newcommand{\tilf}{\widetilde{f}}
\newcommand{\tilF}{\widetilde{F}}
\newcommand{\hatbf}{\widehat{\boldf}}
\newcommand{\hatbF}{\widehat{\bF}}
\newcommand{\tilbf}{\widetilde{\boldf}}
\newcommand{\tilbF}{\widetilde{\bF}}

\newcommand{\hatg}{\widehat{g}}
\newcommand{\hatG}{\widehat{G}}
\newcommand{\tilg}{\widetilde{g}}
\newcommand{\tilG}{\widetilde{G}}
\newcommand{\hatbg}{\widehat{\bg}}
\newcommand{\hatbG}{\widehat{\bG}}
\newcommand{\tilbg}{\widetilde{\bg}}
\newcommand{\tilbG}{\widetilde{\bG}}

\newcommand{\hath}{\widehat{h}}
\newcommand{\hatH}{\widehat{H}}
\newcommand{\tilh}{\widetilde{h}}
\newcommand{\tilH}{\widetilde{H}}
\newcommand{\hatbh}{\widehat{\bh}}
\newcommand{\hatbH}{\widehat{\bH}}
\newcommand{\tilbh}{\widetilde{\bh}}
\newcommand{\tilbH}{\widetilde{\bH}}

\newcommand{\hati}{\widehat{i}}
\newcommand{\hatI}{\widehat{I}}
\newcommand{\tili}{\widetilde{i}}
\newcommand{\tilI}{\widetilde{I}}
\newcommand{\hatbi}{\widehat{\bi}}
\newcommand{\hatbI}{\widehat{\bI}}
\newcommand{\tilbi}{\widetilde{\bi}}
\newcommand{\tilbI}{\widetilde{\bI}}

\newcommand{\hatj}{\widehat{j}}
\newcommand{\hatJ}{\widehat{J}}
\newcommand{\tilj}{\widetilde{j}}
\newcommand{\tilJ}{\widetilde{J}}
\newcommand{\hatbj}{\widehat{\bj}}
\newcommand{\hatbJ}{\widehat{\bJ}}
\newcommand{\tilbj}{\widetilde{\bj}}
\newcommand{\tilbJ}{\widetilde{\bJ}}

\newcommand{\hatk}{\widehat{k}}
\newcommand{\hatK}{\widehat{K}}
\newcommand{\tilk}{\widetilde{k}}
\newcommand{\tilK}{\widetilde{K}}
\newcommand{\hatbk}{\widehat{\bk}}
\newcommand{\hatbK}{\widehat{\bK}}
\newcommand{\tilbk}{\widetilde{\bk}}
\newcommand{\tilbK}{\widetilde{\bK}}

\newcommand{\hatl}{\widehat{l}}
\newcommand{\hatL}{\widehat{L}}
\newcommand{\till}{\widetilde{l}}
\newcommand{\tilL}{\widetilde{L}}
\newcommand{\hatbl}{\widehat{\bl}}
\newcommand{\hatbL}{\widehat{\bL}}
\newcommand{\tilbl}{\widetilde{\bl}}
\newcommand{\tilbL}{\widetilde{\bL}}

\newcommand{\hatm}{\widehat{m}}
\newcommand{\hatM}{\widehat{M}}
\newcommand{\tilm}{\widetilde{m}}
\newcommand{\tilM}{\widetilde{M}}
\newcommand{\hatbm}{\widehat{\boldm}}
\newcommand{\hatbM}{\widehat{\bM}}
\newcommand{\tilbm}{\widetilde{\boldm}}
\newcommand{\tilbM}{\widetilde{\bM}}

\newcommand{\hatn}{\widehat{n}}
\newcommand{\hatN}{\widehat{N}}
\newcommand{\tiln}{\widetilde{n}}
\newcommand{\tilN}{\widetilde{N}}
\newcommand{\hatbn}{\widehat{\bn}}
\newcommand{\hatbN}{\widehat{\bN}}
\newcommand{\tilbn}{\widetilde{\bn}}
\newcommand{\tilbN}{\widetilde{\bN}}

\newcommand{\hato}{\widehat{o}}
\newcommand{\hatO}{\widehat{O}}
\newcommand{\tilo}{\widetilde{o}}
\newcommand{\tilO}{\widetilde{O}}
\newcommand{\hatbo}{\widehat{\bo}}
\newcommand{\hatbO}{\widehat{\bO}}
\newcommand{\tilbo}{\widetilde{\bo}}
\newcommand{\tilbO}{\widetilde{\bO}}

\newcommand{\hatp}{\widehat{p}}
\newcommand{\hatP}{\widehat{P}}
\newcommand{\tilp}{\widetilde{p}}
\newcommand{\tilP}{\widetilde{P}}
\newcommand{\hatbp}{\widehat{\bp}}
\newcommand{\hatbP}{\widehat{\bP}}
\newcommand{\tilbp}{\widetilde{\bp}}
\newcommand{\tilbP}{\widetilde{\bP}}

\newcommand{\hatq}{\widehat{q}}
\newcommand{\hatQ}{\widehat{Q}}
\newcommand{\tilq}{\widetilde{q}}
\newcommand{\tilQ}{\widetilde{Q}}
\newcommand{\hatbq}{\widehat{\bq}}
\newcommand{\hatbQ}{\widehat{\bQ}}
\newcommand{\tilbq}{\widetilde{\bq}}
\newcommand{\tilbQ}{\widetilde{\bQ}}

\newcommand{\hatr}{\widehat{r}}
\newcommand{\hatR}{\widehat{R}}
\newcommand{\tilr}{\widetilde{r}}
\newcommand{\tilR}{\widetilde{R}}
\newcommand{\hatbr}{\widehat{\br}}
\newcommand{\hatbR}{\widehat{\bR}}
\newcommand{\tilbr}{\widetilde{\br}}
\newcommand{\tilbR}{\widetilde{\bR}}

\newcommand{\hats}{\widehat{s}}
\newcommand{\hatS}{\widehat{S}}
\newcommand{\tils}{\widetilde{s}}
\newcommand{\tilS}{\widetilde{S}}
\newcommand{\hatbs}{\widehat{\bs}}
\newcommand{\hatbS}{\widehat{\bS}}
\newcommand{\tilbs}{\widetilde{\bs}}
\newcommand{\tilbS}{\widetilde{\bS}}

\newcommand{\hatt}{\widehat{t}}
\newcommand{\hatT}{\widehat{T}}
\newcommand{\tilt}{\widetilde{t}}
\newcommand{\tilT}{\widetilde{T}}
\newcommand{\hatbt}{\widehat{\bt}}
\newcommand{\hatbT}{\widehat{\bT}}
\newcommand{\tilbt}{\widetilde{\bt}}
\newcommand{\tilbT}{\widetilde{\bT}}

\newcommand{\hatu}{\widehat{u}}
\newcommand{\hatU}{\widehat{U}}
\newcommand{\tilu}{\widetilde{u}}
\newcommand{\tilU}{\widetilde{U}}
\newcommand{\hatbu}{\widehat{\bu}}
\newcommand{\hatbU}{\widehat{\bU}}
\newcommand{\tilbu}{\widetilde{\bu}}
\newcommand{\tilbU}{\widetilde{\bU}}

\newcommand{\hatv}{\widehat{v}}
\newcommand{\hatV}{\widehat{V}}
\newcommand{\tilv}{\widetilde{v}}
\newcommand{\tilV}{\widetilde{V}}
\newcommand{\hatbv}{\widehat{\bv}}
\newcommand{\hatbV}{\widehat{\bV}}
\newcommand{\tilbv}{\widetilde{\bv}}
\newcommand{\tilbV}{\widetilde{\bV}}

\newcommand{\hatw}{\widehat{w}}
\newcommand{\hatW}{\widehat{W}}
\newcommand{\tilw}{\widetilde{w}}
\newcommand{\tilW}{\widetilde{W}}
\newcommand{\hatbw}{\widehat{\bw}}
\newcommand{\hatbW}{\widehat{\bW}}
\newcommand{\tilbw}{\widetilde{\bw}}
\newcommand{\tilbW}{\widetilde{\bW}}

\newcommand{\hatx}{\widehat{x}}
\newcommand{\hatX}{\widehat{X}}
\newcommand{\tilx}{\widetilde{x}}
\newcommand{\tilX}{\widetilde{X}}
\newcommand{\hatbx}{\widehat{\bx}}
\newcommand{\hatbX}{\widehat{\bX}}
\newcommand{\tilbx}{\widetilde{\bx}}
\newcommand{\tilbX}{\widetilde{\bX}}

\newcommand{\haty}{\widehat{y}}
\newcommand{\hatY}{\widehat{Y}}
\newcommand{\tily}{\widetilde{y}}
\newcommand{\tilY}{\widetilde{Y}}
\newcommand{\hatby}{\widehat{\by}}
\newcommand{\hatbY}{\widehat{\bY}}
\newcommand{\tilby}{\widetilde{\by}}
\newcommand{\tilbY}{\widetilde{\bY}}

\newcommand{\hatz}{\widehat{z}}
\newcommand{\hatZ}{\widehat{Z}}
\newcommand{\tilz}{\widetilde{z}}
\newcommand{\tilZ}{\widetilde{Z}}
\newcommand{\hatbz}{\widehat{\bz}}
\newcommand{\hatbZ}{\widehat{\bZ}}
\newcommand{\tilbz}{\widetilde{\bz}}
\newcommand{\tilbZ}{\widetilde{\bZ}}

\newcommand{\bara}{\overline{a}}
\newcommand{\barb}{\overline{b}}
\newcommand{\barc}{\overline{c}}
\newcommand{\bard}{\overline{d}}
\newcommand{\bare}{\overline{e}}
\newcommand{\barf}{\overline{f}}
\newcommand{\barg}{\overline{g}}
\newcommand{\barh}{\overline{h}}
\newcommand{\bari}{\overline{i}}
\newcommand{\barj}{\overline{j}}
\newcommand{\bark}{\overline{k}}
\newcommand{\barl}{\overline{l}}
\newcommand{\barm}{\overline{m}}
\newcommand{\barn}{\overline{n}}
\newcommand{\baro}{\overline{o}}
\newcommand{\barp}{\overline{p}}
\newcommand{\barq}{\overline{q}}
\newcommand{\barr}{\overline{r}}
\newcommand{\bars}{\overline{s}}
\newcommand{\bart}{\overline{t}}
\newcommand{\baru}{\overline{u}}
\newcommand{\barv}{\overline{v}}
\newcommand{\barw}{\overline{w}}
\newcommand{\barx}{\overline{x}}
\newcommand{\bary}{\overline{y}}
\newcommand{\barz}{\overline{z}}

\newcommand{\barA}{\overline{A}}
\newcommand{\barB}{\overline{B}}
\newcommand{\barC}{\overline{C}}
\newcommand{\barD}{\overline{D}}
\newcommand{\barE}{\overline{E}}
\newcommand{\barF}{\overline{F}}
\newcommand{\barG}{\overline{G}}
\newcommand{\barH}{\overline{H}}
\newcommand{\barI}{\overline{I}}
\newcommand{\barJ}{\overline{J}}
\newcommand{\barK}{\overline{K}}
\newcommand{\barL}{\overline{L}}
\newcommand{\barM}{\overline{M}}
\newcommand{\barN}{\overline{N}}
\newcommand{\barO}{\overline{O}}
\newcommand{\barP}{\overline{P}}
\newcommand{\barQ}{\overline{Q}}
\newcommand{\barR}{\overline{R}}
\newcommand{\barS}{\overline{S}}
\newcommand{\barT}{\overline{T}}
\newcommand{\barU}{\overline{U}}
\newcommand{\barV}{\overline{V}}
\newcommand{\barW}{\overline{W}}
\newcommand{\barX}{\overline{X}}
\newcommand{\barY}{\overline{Y}}
\newcommand{\barZ}{\overline{Z}}





% Bold greek
\newcommand{\balpha}{\bm{\alpha}}
\newcommand{\bbeta}{\bm{\beta}}
\newcommand{\bgamma}{\bm{\gamma}}
\newcommand{\bdelta}{\bm{\delta}}
\newcommand{\btheta}{\bm{\theta}}
\newcommand{\btau}{\bm{\tau}}
\newcommand{\bpi}{\bm{\pi}}
\newcommand{\bepsilon}{\bm{\epsilon}}
\newcommand{\veps}{\varepsilon}
\newcommand{\bvarepsilon}{\bm{\varepsilon}}
\newcommand{\bsigma}{\bm{\sigma}}
\newcommand{\bzeta}{\bm{\zeta}}
\newcommand{\bmeta}{\bm{\eta}}
\newcommand{\bkappa}{\bm{\kappa}}
\newcommand{\bchi}{\bm{\chi}}
\newcommand{\bphi}{\bm{\phi}}
\newcommand{\bpsi}{\bm{\psi}}
\newcommand{\bomega}{\bm{\omega}}
\newcommand{\bxi}{\bm{\xi}}
\newcommand{\blambda}{\bm{\lambda}}
\newcommand{\brho}{\bm{\rho}}

\newcommand{\bGamma}{\bm{\Gamma}}
\newcommand{\bLambda}{\bm{\Lambda}}
\newcommand{\bSigma	}{\bm{\Sigma}}
\newcommand{\bPsi}{\bm{\Psi}}
\newcommand{\bDelta}{\bm{\Delta}}
\newcommand{\bXi}{\bm{\Xi}}
\newcommand{\bUpsilon}{\bm{\Upsilon}}
\newcommand{\bOmega}{\bm{\Omega}}
\newcommand{\bPhi}{\bm{\Phi}}
\newcommand{\bPi}{\bm{\Pi}}
\newcommand{\bTheta}{\bm{\Theta}}

\newcommand{\talpha}{\widetilde{\alpha}}
\newcommand{\tbeta}{\widetilde{\beta}}
\newcommand{\tgamma}{\widetilde{\gamma}}
\newcommand{\tdelta}{\widetilde{\delta}}
\newcommand{\ttheta}{\widetilde{\theta}}
\newcommand{\ttau}{\widetilde{\tau}}
\newcommand{\tpi}{\widetilde{\pi}}
\newcommand{\tepsilon}{\widetilde{\epsilon}}
\newcommand{\tvarepsilon}{\widetilde{\varepsilon}}
\newcommand{\tsigma}{\widetilde{\sigma}}
\newcommand{\tzeta}{\widetilde{\zeta}}
\newcommand{\tmeta}{\widetilde{\eta}}
\newcommand{\tkappa}{\widetilde{\kappa}}
\newcommand{\tchi}{\widetilde{\chi}}
\newcommand{\tphi}{\widetilde{\phi}}
\newcommand{\tpsi}{\widetilde{\psi}}
\newcommand{\tomega}{\widetilde{\omega}}
\newcommand{\txi}{\widetilde{\xi}}
\newcommand{\tlambda}{\widetilde{\lambda}}
\newcommand{\trho}{\widetilde{\rho}}
\newcommand{\tnu}{\widetilde{\nu}}

\newcommand{\tDelta}{\widetilde{\Delta}}

\newcommand{\halpha}{\widehat{\alpha}}
\newcommand{\hbeta}{\widehat{\beta}}
\newcommand{\hgamma}{\widehat{\gamma}}
\newcommand{\hdelta}{\widehat{\delta}}
\newcommand{\htheta}{\widehat{\theta}}
\newcommand{\htau}{\widehat{\tau}}
\newcommand{\hpi}{\widehat{\pi}}
\newcommand{\hepsilon}{\widehat{\epsilon}}
\newcommand{\hvarepsilon}{\widehat{\varepsilon}}
\newcommand{\hsigma}{\widehat{\sigma}}
\newcommand{\hzeta}{\widehat{\zeta}}
\newcommand{\hmeta}{\widehat{\eta}}
\newcommand{\hkappa}{\widehat{\kappa}}
\newcommand{\hchi}{\widehat{\chi}}
\newcommand{\hphi}{\widehat{\phi}}
\newcommand{\hpsi}{\widehat{\psi}}
\newcommand{\homega}{\widehat{\omega}}
\newcommand{\hxi}{\widehat{\xi}}
\newcommand{\hlambda}{\widehat{\lambda}}
\newcommand{\hrho}{\widehat{\rho}}

% Define underlined letters
\newcommand{\ua}{\underline{a}}
\newcommand{\uA}{\underline{A}}
\newcommand{\ub}{\underline{b}}
\newcommand{\uB}{\underline{B}}
\newcommand{\uc}{\underline{c}}
\newcommand{\uC}{\underline{C}}
\newcommand{\ud}{\underline{d}}
\newcommand{\uD}{\underline{D}}
\newcommand{\ue}{\underline{e}}
\newcommand{\uE}{\underline{E}}
\newcommand{\uf}{\underline{f}}
\newcommand{\uF}{\underline{F}}
\newcommand{\ug}{\underline{g}}
\newcommand{\uG}{\underline{G}}
\newcommand{\uh}{\underline{h}}
\newcommand{\uH}{\underline{H}}
\newcommand{\ui}{\underline{i}}
\newcommand{\uI}{\underline{I}}
\newcommand{\uj}{\underline{j}}
\newcommand{\uJ}{\underline{J}}
\newcommand{\uk}{\underline{k}}
\newcommand{\uK}{\underline{K}}
\newcommand{\ul}{\underline{l}}
\newcommand{\uL}{\underline{L}}
\newcommand{\um}{\underline{m}}
\newcommand{\uM}{\underline{M}}
\newcommand{\un}{\underline{n}}
\newcommand{\uN}{\underline{N}}
\newcommand{\uo}{\underline{o}}
\newcommand{\uO}{\underline{O}}
\newcommand{\up}{\underline{p}}
\newcommand{\uP}{\underline{P}}
\newcommand{\uq}{\underline{q}}
\newcommand{\uQ}{\underline{Q}}
\newcommand{\ur}{\underline{r}}
\newcommand{\uR}{\underline{R}}
\newcommand{\us}{\underline{s}}
\newcommand{\uS}{\underline{S}}
\newcommand{\ut}{\underline{t}}
\newcommand{\uT}{\underline{T}}
\newcommand{\uu}{\underline{u}}
\newcommand{\uU}{\underline{U}}
\newcommand{\uv}{\underline{v}}
\newcommand{\uV}{\underline{V}}
\newcommand{\uw}{\underline{w}}
\newcommand{\uW}{\underline{W}}
\newcommand{\ux}{\underline{x}}
\newcommand{\uX}{\underline{X}}
\newcommand{\uy}{\underline{y}}
\newcommand{\uY}{\underline{Y}}
\newcommand{\uz}{\underline{z}}
\newcommand{\uZ}{\underline{Z}}


% functional dot

\def\fndot{\, \cdot \,}
\def\fndia{\, \diamond \,}
\def\fnstar{\, \star \,}

% exponential

\newcommand{\eexp}[1]{e^{#1}}

% iid

\newcommand{\iid}{i.i.d.\ }

% convergence

\newcommand{\convp}{\stackrel{\mathrm{p}}{\longrightarrow}}
\newcommand{\convwpone}{\stackrel{\mathrm{w.p.1}}{\longrightarrow}}
\newcommand{\convas}{\xrightarrow{\mathrm{a.s.}}}
\newcommand{\convd}{\stackrel{\mathrm{d}}{\longrightarrow}}
\newcommand{\convD}{\stackrel{\mathrm{D}}{\longrightarrow}}

\newcommand{\ceil}[1]{\lceil{#1}\rceil}
\newcommand{\floor}[1]{\lfloor{#1}\rfloor}
\newcommand{\lrangle}[2]{\left\langle{#1},{#2}\right\rangle}


\newcommand{\dotleq}{\stackrel{.}{\leq}}
\newcommand{\dotlt}{\stackrel{.}{<}}
\newcommand{\dotgeq}{\stackrel{.}{\geq}}
\newcommand{\dotgt}{\stackrel{.}{>}}
\newcommand{\dotdoteq}{\stackrel{\,..}{=}}
% Inequalities
\newcommand{\eqa}{\stackrel{(a)}{=}}
\newcommand{\eqb}{\stackrel{(b)}{=}}
\newcommand{\eqc}{\stackrel{(c)}{=}}
\newcommand{\eqd}{\stackrel{(d)}{=}}
\newcommand{\eqe}{\stackrel{(e)}{=}}
\newcommand{\eqf}{\stackrel{(f)}{=}}
\newcommand{\eqg}{\stackrel{(g)}{=}}
\newcommand{\eqh}{\stackrel{(h)}{=}}
\newcommand{\lea}{\stackrel{(a)}{\le}}
\newcommand{\leb}{\stackrel{(b)}{\le}}
\newcommand{\lec}{\stackrel{(c)}{\le}}
\newcommand{\led}{\stackrel{(d)}{\le}}
\newcommand{\lee}{\stackrel{(e)}{\le}}
\newcommand{\lef}{\stackrel{(f)}{\le}}
\newcommand{\leg}{\stackrel{(g)}{\le}}
\newcommand{\leh}{\stackrel{(h)}{\le}}
\newcommand{\gea}{\stackrel{(a)}{\ge}}
\newcommand{\geb}{\stackrel{(b)}{\ge}}
\newcommand{\gec}{\stackrel{(c)}{\ge}}
\newcommand{\ged}{\stackrel{(d)}{\ge}}
\newcommand{\gee}{\stackrel{(e)}{\ge}}
\newcommand{\gef}{\stackrel{(f)}{\ge}}
\newcommand{\geg}{\stackrel{(g)}{\ge}}
\newcommand{\geh}{\stackrel{(h)}{\ge}}
 
\newcommand{\Pen}{P_{\mathrm{e}}^{(n)}}
\newcommand{\Penone}{P_{\mathrm{e}, 1}^{(n)}}
\newcommand{\Pentwo}{P_{\mathrm{e}, 2}^{(n)}}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}
\DeclareMathOperator*{\argsup}{arg\,sup}
\DeclareMathOperator*{\arginf}{arg\,inf}
\DeclareMathOperator{\minimize}{minimize}
\DeclareMathOperator{\maximize}{maximize}
%\DeclareMathOperator{\st}{subject\,\,to}
\DeclareMathOperator{\st}{s.t.\;}
\DeclareMathOperator{\erfc}{erfc}
\DeclareMathOperator{\diag}{diag}
\DeclareMathOperator{\cum}{cum}
\DeclareMathOperator{\sgn}{sgn}
\DeclareMathOperator{\tr}{tr}
\DeclareMathOperator{\spn}{span}
\DeclareMathOperator{\supp}{supp}
\DeclareMathOperator{\adj}{adj}
\DeclareMathOperator{\var}{\mathsf{Var}}
\DeclareMathOperator{\Vol}{Vol}
\DeclareMathOperator{\cov}{\mathsf{Cov}}
\DeclareMathOperator{\sech}{sech}
\DeclareMathOperator{\sinc}{sinc}
\DeclareMathOperator{\rank}{rank}
\DeclareMathOperator{\poly}{poly}
\DeclareMathOperator{\vect}{vec}
\newcommand{\Hb}{H_{\mathrm{b}}}%binary entropy
\newcommand{\Ber}{\mathrm{Bern}}
\DeclareMathOperator*{\lms}{l.i.m.\,}
\newcommand{\varop}[1]{\var\left[{#1}\right]}
\newcommand{\covop}[2]{\cov\left({#1},{#2}\right)} 
\newcommand{\bzero}{\mathbf{0}}
\newcommand{\bone}{\mathbf{1}}

\def\independenT#1#2{\mathrel{\rlap{$#1#2$}\mkern5mu{#1#2}}}
\newcommand\indep{\protect\mathpalette{\protect\independenT}{\perp}}
 

\newtheorem{theorem}{Theorem} 
\newtheorem*{theorem*}{Theorem}
\newtheorem{lemma}{Lemma}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{prop}[theorem]{Proposition}
\newtheorem{corollary}{Corollary}
\newtheorem*{assump}{Assumptions} 
\newtheorem{example}{Example} 
\newtheorem{exercise}{Exercise} 

\newtheorem{fact}{Fact}

\theoremstyle{definition}
\newtheorem{definition}{Definition} 

\theoremstyle{remark}
\newtheorem{remark}{Remark}

%\theoremstyle{remark}
%\newtheorem*{psketch}{Proof Sketch}
\newenvironment{psketch}{\noindent{\em Proof Sketch.}\hspace*{1em}}{\qed\bigskip\\}

\renewcommand\qedsymbol{$\square$}

%\newenvironment{proof}[1][Proof]{\begin{trivlist}
%\item[\hskip \labelsep {\bfseries #1}]}{{\hfill$\square$}\end{trivlist}}
%\newenvironment{psketch}[1][Proof Sketch]{\begin{trivlist}
%\item[\hskip \labelsep {\bfseries #1}]}{{\hfill$\square$}\end{trivlist}}
%\newenvironment{definition}[1][Definition]{\begin{trivlist}
%\item[\hskip \labelsep {\bfseries #1}]}{\end{trivlist}}
%\newenvironment{example}[1][Example]{\begin{trivlist}
%\item[\hskip \labelsep {\bfseries #1}]}{\end{trivlist}}
%\newenvironment{remark}[1][Remark]{\begin{trivlist}
%\item[\hskip \labelsep {\bfseries #1}]}{\end{trivlist}}
%\newenvironment{assump}[1][Assumption]{\begin{trivlist}
%\item[\hskip \labelsep {\bfseries #1}]}{\end{trivlist}}

%\newcommand{\qed}{\hfill$\square$}
\newcommand{\qednew}{\nobreak \ifvmode \relax \else
      \ifdim\lastskip<1.5em \hskip-\lastskip
      \hskip1.5em plus0em minus0.5em \fi \nobreak
      \vrule height0.75em width0.5em depth0.25em\fi}

merged.bbl
% Generated by IEEEtran.bst, version: 1.13 (2008/09/30)
\begin{thebibliography}{100}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
  \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl

\bibitem{Zhao_16}
R.~Zhao and V.~Y.~F. Tan, ``Online nonnegative matrix factorization with
  outliers,'' in \emph{Proc.\ ICASSP}, Shanghai, China, Mar. 2016, pp.
  2662--2666.

\bibitem{Tsuge_01}
S.~Tsuge, M.~Shishibori, S.~Kuroiwa, and K.~Kita, ``Dimensionality reduction
  using non-negative matrix factorization for information retrieval,'' in
  \emph{Proc.\ SMC}, vol.~2, Tucson, Arizona, USA, Oct. 2001, pp. 960--965.

\bibitem{Lee_99}
D.~D. Lee and H.~S. Seung, ``Learning the parts of objects by nonnegative
  matrix factorization,'' \emph{Nature}, vol. 401, pp. 788--791, October 1999.

\bibitem{Lee_00}
------, ``Algorithms for non-negative matrix factorization,'' in \emph{Proc.\
  NIPS}, Denver, USA, Dec. 2000, pp. 556--562.

\bibitem{Kim_08a}
J.~Kim and H.~Park, ``Toward faster nonnegative matrix factorization: A new
  algorithm and comparisons,'' in \emph{Proc.\ ICDM}, Pisa, Italy, Dec. 2008,
  pp. 353--362.

\bibitem{Lin_07a}
C.-J. Lin, ``Projected gradient methods for nonnegative matrix factorization,''
  \emph{Neural Comput.}, vol.~19, no.~10, pp. 2756--2779, Oct. 2007.

\bibitem{Kim_08b}
H.~Kim and H.~Park, ``Nonnegative matrix factorization based on alternating
  nonnegativity constrained least squares and active set method,'' \emph{SIAM
  J. Matrix Anal. A.}, vol.~30, no.~2, pp. 713–--730, 2008.

\bibitem{Xu_12}
Y.~Xu, W.~Yin, Z.~Wen, and Y.~Zhang, ``An alternating direction algorithm for
  matrix completion with nonnegative factors,'' \emph{Frontiers Math. China},
  vol.~7, pp. 365–--384, 2012.

\bibitem{Ding_05}
C.~Ding, X.~He, and H.~D. Simon, ``On the equivalence of nonnegative matrix
  factorization and spectral clustering,'' in \emph{Proc.\ SDM}, Newport Beach,
  California, USA, Apr. 2005, pp. 606--610.

\bibitem{Yuan_15}
Y.~Yuan, Y.~Feng, and X.~Lu, ``Projection-based nmf for hyperspectral
  unmixing,'' \emph{IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.}, vol.~8,
  no.~6, pp. 2632--2643, Jun. 2015.

\bibitem{Durrieu_11}
J.~L. Durrieu, B.~David, and G.~Richard, ``A musically motivated mid-level
  representation for pitch estimation and musical audio source separation,''
  \emph{IEEE J. Sel. Top. Signal Process.}, vol.~5, no.~6, pp. 1180--1191, Oct.
  2011.

\bibitem{Cao_07}
B.~Cao, D.~Shen, J.-T. Sun, X.~Wang, Q.~Yang, and Z.~Chen, ``Detect and track
  latent factors with online nonnegative matrix factorization,'' in
  \emph{Proc.\ IJCAI}, Hyderabad, India, Jan. 2007, pp. 2689--2694.

\bibitem{Zhang_11}
L.~Zhang, Z.~Chen, M.~Zheng, and X.~He, ``Robust nonnegative matrix
  factorization,'' \emph{Frontiers Elect. Electron. Eng. China}, vol.~6, no.~2,
  pp. 192--200, 2011.

\bibitem{RAD_15}
Netflix, ``Rad - outlier detection on big data,''
  \url{http://techblog.netflix.com/2015/02/rad-outlier-detection-on-big-data.html},
  2015.

\bibitem{Bucak_08}
S.~S. Bucak and B.~Gunsel, ``Incremental subspace learning via non-negative
  matrix factorization,'' \emph{Pattern Recogn.}, vol.~42, no.~5, pp.
  788–--797, May 2009.

\bibitem{Guan_12}
N.~Guan, D.~Tao, Z.~Luo, and B.~Yuan, ``Online nonnegative matrix factorization
  with robust stochastic approximation,'' \emph{IEEE Trans. Neural Netw. Learn.
  Syst.}, vol.~23, no.~7, pp. 1087--1099, Jul. 2012.

\bibitem{Lef_11}
A.~Lef\`evre, F.~Bach, and C.~F\'evotte, ``Online algorithms for nonnegative
  matrix factorization with the itakura-saito divergence,'' in \emph{Proc.\
  WASPAA}, New Paltz, New York, USA, Oct 2011, pp. 313--316.

\bibitem{WangFei_11}
F.~Wang, C.~Tan, A.~C. Konig, and P.~Li, ``Efficient document clustering via
  online nonnegative matrix factorizations,'' in \emph{Proc.\ SDM}, Mesa,
  Arizona, USA, Apr. 2011, pp. 908--919.

\bibitem{WuShen_14}
Y.~Wu, B.~Shen, and H.~Ling, ``Visual tracking via online nonnegative matrix
  factorization,'' \emph{IEEE Trans. Circuits Syst. Video Technol.}, vol.~24,
  no.~3, pp. 374--383, Mar. 2014.

\bibitem{Liu_10}
C.~Liu, H.~chih Yang, J.~Fan, L.-W. He, and Y.-M. Wang, ``Distributed
  nonnegative matrix factorization for web-scale dyadic data analysis on
  mapreduce,'' in \emph{Proc.\ WWW}, Raleigh, North Carolina, USA, Apr. 2010,
  pp. 681--690.

\bibitem{Gemulla_11}
R.~Gemulla, P.~J. Haas, Y.~Sismanis, C.~Teflioudi, and F.~Makari, ``Large-scale
  matrix factorization with distributed stochastic gradient descent,'' in
  \emph{Proc.\ KDD}, San Diego, California, USA, Aug. 2011, pp. 69--77.

\bibitem{Du_14}
S.~S. Du, Y.~Liu, B.~Chen, and L.~Li, ``Maxios: Large scale nonnegative matrix
  factorization for collaborative filtering,'' in \emph{Proc.\ NIPS-DMLMC},
  Montr\'eal, Canada, Dec. 2014.

\bibitem{Chen_15}
J.~Chen, Z.~J. Towfic, and A.~H. Sayed, ``Dictionary learning over distributed
  models,'' \emph{IEEE Trans. Signal Process.}, vol.~63, no.~4, pp. 1001--1016,
  2015.

\bibitem{Halko_11}
N.~Halko, P.~G. Martinsson, and J.~A. Tropp, ``Finding structure with
  randomness: Probabilistic algorithms for constructing approximate matrix
  decompositions,'' \emph{SIAM Rev.}, vol.~53, no.~2, pp. 217--–288, 2011.

\bibitem{Tepper_16}
M.~Tepper and G.~Sapiro, ``Compressed nonnegative matrix factorization is fast
  and accurate,'' \emph{IEEE Trans. Signal Process.}, vol.~64, no.~9, pp.
  2269--2283, May 2016.

\bibitem{Gao_15}
H.~Gao, F.~Nie, W.~Cai, and H.~Huang, ``Robust capped norm nonnegative matrix
  factorization: Capped norm {NMF},'' in \emph{Proc.\ CIKM}, Melbourne,
  Australia, Oct. 2015, pp. 871--880.

\bibitem{Huang_14}
J.~Huang, F.~Nie, H.~Huang, and C.~Ding, ``Robust manifold nonnegative matrix
  factorization,'' \emph{ACM Trans. Knowl. Discov. Data}, vol.~8, no.~3, pp.
  1--21, Jun. 2014.

\bibitem{Yang_13}
S.~Yang, C.~Hou, C.~Zhang, and Y.~Wu, ``Robust non-negative matrix
  factorization via joint sparse and graph regularization for transfer
  learning,'' \emph{Neural Comput. and Appl.}, vol.~23, no.~2, pp. 541--559,
  2013.

\bibitem{Du_12}
L.~Du, X.~Li, and Y.-D. Shen, ``Robust nonnegative matrix factorization via
  half-quadratic minimization,'' in \emph{Proc.\ ICDM}, Brussels, Belgium, Dec.
  2012, pp. 201--210.

\bibitem{Ding_12}
C.~Ding and D.~Kong, ``Nonnegative matrix factorization using a robust error
  function,'' in \emph{Proc.\ ICASSP}, Kyoto, Japan, Mar. 2012, pp. 2033--2036.

\bibitem{Ding_11}
D.~Kong, C.~Ding, and H.~Huang, ``Robust nonnegative matrix factorization using
  $\ell_{21}$-norm,'' in \emph{Proc.\ CIKM}, Glasgow, Scotland, UK, Oct. 2011,
  pp. 673--682.

\bibitem{Cichoc_11}
A.~Cichocki, S.~Cruces, and S.-i. Amari, ``Generalized alpha-beta divergences
  and their application to robust nonnegative matrix factorization,''
  \emph{Entropy}, vol.~13, no.~1, pp. 134--170, 2011.

\bibitem{Kasi_12}
S.~P. Kasiviswanathan, H.~Wang, A.~Banerjee, and P.~Melville, ``Online
  $\ell_1$-dictionary learning with application to novel document detection,''
  in \emph{Proc.\ NIPS}, Lake Tahoe, USA, Dec. 2012, pp. 2258--2266.

\bibitem{Fev_14}
C.~F\'evotte and N.~Dobigeon, ``Nonlinear hyperspectral unmixing with robust
  nonnegative matrix factorization,'' \emph{IEEE Trans. Image Process.},
  vol.~24, no.~12, pp. 4810--4819, 2015.

\bibitem{Shen_14}
B.~Shen, B.~Liu, Q.~Wang, and R.~Ji, ``Robust nonnegative matrix factorization
  via $l_1$ norm regularization by multiplicative updating rules,'' in
  \emph{Proc.\ ICIP}, Paris, France, Oct. 2014, pp. 5282--5286.

\bibitem{Feng_13}
J.~Feng, H.~Xu, and S.~Yan, ``Online robust {PCA} via stochastic
  optimization,'' in \emph{Proc.\ NIPS}, Lake Tahoe, USA, Dec. 2013, pp.
  404--412.

\bibitem{Shen_14b}
J.~Shen, H.~Xu, and P.~Li, ``Online optimization for max-norm regularization,''
  in \emph{Proc.\ NIPS}, Montr\'eal, Canada, Dec. 2014, pp. 1718--1726.

\bibitem{Wang_13}
N.~Wang, J.~Wang, and D.-Y. Yeung, ``Online robust non-negative dictionary
  learning for visual tracking,'' in \emph{Proc.\ ICCV}, Sydney, Australia,
  Dec. 2013, pp. 657--664.

\bibitem{Rock_70}
R.~T. Rockafellar, \emph{Convex analysis}.\hskip 1em plus 0.5em minus
  0.4em\relax Princeton University Press, 1970.

\bibitem{Vaart_00}
A.~W. van~der Vaart, \emph{Asymptotic Statistics}.\hskip 1em plus 0.5em minus
  0.4em\relax Cambridge Press, 2000.

\bibitem{Met_82}
M.~M\'{e}tivier, \emph{Semimartingales: A Course on Stochastic
  Processes}.\hskip 1em plus 0.5em minus 0.4em\relax de Gruyter, 1982.

\bibitem{Paa_94}
P.~Paatero and U.~Tapper, ``Positive matrix factorization: A non-negative
  factor model with optimal utilization of error estimates of data values,''
  \emph{Environmetrics}, vol.~5, no.~2, pp. 111--126, 1994.

\bibitem{Hoyer_04}
P.~O. Hoyer, ``Non-negative matrix factorization with sparseness constraints,''
  \emph{J. Mach. Learn. Res.}, vol.~5, pp. 1457--1469, Dec. 2004.

\bibitem{Mairal_10}
J.~Mairal, F.~Bach, J.~Ponce, and G.~Sapiro, ``Online learning for matrix
  factorization and sparse coding,'' \emph{J. Mach. Learn. Res.}, vol.~11, pp.
  19--60, Mar 2010.

\bibitem{WangDong_13}
D.~Wang and H.~Lu, ``On-line learning parts-based representation via
  incremental orthogonal projective non-negative matrix factorization,''
  \emph{Signal Process.}, vol.~93, no.~6, pp. 1608 -- 1623, 2013.

\bibitem{Xing_13}
J.~Xing, J.~Gao, B.~Li, W.~Hu, and S.~Yan, ``Robust object tracking with online
  multi-lifespan dictionary learning,'' in \emph{Proc.\ ICCV}, Sydney,
  Australia, Dec. 2013, pp. 665--672.

\bibitem{Zhang_15}
S.~Zhang, S.~Kasiviswanathan, P.~C. Yuen, and M.~Harandi, ``Online dictionary
  learning on symmetric positive definite manifolds with vision applications,''
  in \emph{Proc.\ AAAI}, Austin, Texas, Jan. 2015, pp. 3165--3173.

\bibitem{Zhang_15b}
X.~Zhang, N.~Guan, D.~Tao, X.~Qiu, and Z.~Luo, ``Online multi-modal robust
  non-negative dictionary learning for visual tracking,'' \emph{PLoS ONE},
  vol.~10, no.~5, pp. 1--17, 2015.

\bibitem{Zhan_16}
J.~Zhan, B.~Lois, and N.~Vaswani, ``Online (and offline) robust {PCA}: Novel
  algorithms and performance guarantees,'' arXiv:1601.07985, 2016.

\bibitem{Lois_15}
B.~Lois and N.~Vaswani, ``Online matrix completion and online robust {PCA},''
  arXiv:1503.03525, 2015.

\bibitem{Guo_15}
X.~Guo, ``Online robust low rank matrix recovery,'' in \emph{Proc.\ IJCAI},
  Buenos Aires, Argentina, Jul. 2015, pp. 3540--3546.

\bibitem{Spre_12}
P.~Sprechmann, A.~Bronstein, and G.~Sapiro, ``Real-time online singing voice
  separation from monaural recordings using robust low-rank modeling,'' in
  \emph{Proc.\ ISMIR}, Porto, Portugal, Oct. 2012, pp. 67--72.

\bibitem{Spre_12b}
P.~Sprechmann, A.~M. Bronstein, and G.~Sapiro, ``Learning efficient sparse and
  low rank models,'' arXiv:1212.3631, 2012.

\bibitem{Tsypkin_71}
Y.~Tsypkin, \emph{Adaptation and Learning in automatic systems}.\hskip 1em plus
  0.5em minus 0.4em\relax Academic Press, New York, 1971.

\bibitem{Bottou_98}
L.~Bottou, ``Online algorithms and stochastic approximations,'' in \emph{Online
  Learning and Neural Networks}.\hskip 1em plus 0.5em minus 0.4em\relax
  Cambridge University Press, 1998.

\bibitem{Bottou_08}
L.~Bottou and O.~Bousquet, ``The tradeoffs of large scale learning,'' in
  \emph{Proc.\ NIPS}, Vancouver, Canada, Dec. 2008, pp. 161--168.

\bibitem{Shap_07}
A.~Shapiro and A.~Philpott, ``A tutorial on stochastic programming,''
  http://stoprog.org/sites/default/files/SPTutorial/TutorialSP.pdf, 2007.

\bibitem{Lin_07b}
C.-J. Lin, ``On the convergence of multiplicative update algorithms for
  nonnegative matrix factorization,'' \emph{IEEE Trans. Neural Netw.}, vol.~18,
  no.~6, pp. 1589--1596, 2007.

\bibitem{Mensch_16}
A.~Mensch, J.~Mairal, B.~Thirion, and G.~Varoquaux, ``Dictionary learning for
  massive matrix factorization,'' in \emph{Proc.\ ICML}, New York, USA, Jul.
  2016.

\bibitem{Mairal_13}
J.~Mairal, ``Stochastic majorization-minimization algorithms for large-scale
  optimization,'' in \emph{Proc.\ NIPS}, Lake Tahoe, USA, Dec. 2013, pp.
  2283--2291.

\bibitem{Raza_16}
M.~Razaviyayn, M.~Sanjabi, and Z.-Q. Luo, ``A stochastic successive
  minimization method for nonsmooth nonconvex optimization with applications to
  transceiver design in wireless communication networks,'' \emph{Math.
  Program.}, vol. 157, no.~2, pp. 515--545, 2016.

\bibitem{Sun_14}
D.~Sun and C.~Fevotte, ``Alternating direction method of multipliers for
  non-negative matrix factorization with the beta-divergence,'' in \emph{Proc.\
  ICASSP}, Florence, Italy, May 2014, pp. 6201--6205.

\bibitem{Raza_13}
M.~Razaviyayn, M.~Hong, and Z.-Q. Luo, ``A unified convergence analysis of
  block successive minimization methods for nonsmooth optimization,''
  \emph{SIAM J. Optim.}, vol.~23, no.~2, pp. 1126–--1153, 2013.

\bibitem{Hong_16}
M.~Hong, M.~Razaviyayn, Z.~Q. Luo, and J.~S. Pang, ``A unified algorithmic
  framework for block-structured optimization involving big data: With
  applications in machine learning and signal processing,'' \emph{IEEE Signal
  Process. Mag.}, vol.~33, no.~1, pp. 57--77, Jan. 2016.

\bibitem{ZhangBox_15}
H.~Zhang and L.~Cheng, ``Projected shrinkage algorithm for box-constrained
  $\ell_1$-minimization,'' \emph{Optim. Lett.}, vol.~1, no.~1, pp. 1–--16,
  2015.

\bibitem{Tseng_08}
P.~Tseng, ``On accelerated proximal gradient methods for convex-concave
  optimization,'' Technical report, University of Washington, Seattle, 2008.

\bibitem{Nest_13}
Y.~Nesterov, ``Gradient methods for minimizing composite functions,''
  \emph{Math. Program.}, vol. 140, no.~1, pp. 125--161, 2013.

\bibitem{Vavasis_09}
S.~A. Vavasis, ``On the complexity of nonnegative matrix factorization,''
  \emph{SIAM J. Optim.}, vol.~20, no.~3, pp. 1364--–1377, 2009.

\bibitem{Haji_16}
D.~Hajinezhad, T.~H. Chang, X.~Wang, Q.~Shi, and M.~Hong, ``Nonnegative matrix
  factorization using admm: Algorithm and convergence analysis,'' in
  \emph{Proc.\ ICASSP}, Shanghai, China, Mar. 2016, pp. 4742--4746.

\bibitem{Donoho_04}
D.~Donoho and V.~Stodden, ``When does non-negative matrix factorization give
  correct decomposition into parts?'' in \emph{Proc.\ NIPS}, Vancouver, Canada,
  Dec. 2004, pp. 1141--1148.

\bibitem{Cohen_93}
J.~E. Cohen and U.~G. Rothblum, ``Nonnegative ranks, decompositions, and
  factorizations of nonnegative matrices,'' \emph{Linear Algebra Appl.}, vol.
  190, no.~1, pp. 149 -- 168, 1993.

\bibitem{Arora_12}
S.~Arora, R.~Ge, R.~Kannan, and A.~Moitra, ``Computing a nonnegative matrix
  factorization -- provably,'' in \emph{Proc.\ STOC}, New York, New York, USA,
  May 2012, pp. 145--162.

\bibitem{Gillis_12}
N.~Gillis, ``Sparse and unique nonnegative matrix factorization through data
  preprocessing,'' \emph{J. Mach. Learn. Res.}, vol.~13, no.~1, pp. 3349--3386,
  Nov. 2012.

\bibitem{Vanda_16}
A.~Vandaele, N.~Gillis, F.~Glineur, and D.~Tuyttens, ``Heuristics for exact
  nonnegative matrix factorization,'' \emph{J. Glob. Optim.}, vol.~65, no.~2,
  pp. 369--400, 2016.

\bibitem{Nemi_09}
A.~Nemirovski, A.~Juditsky, G.~Lan, and A.~Shapiro, ``Robust stochastic
  approximation approach to stochastic programming,'' \emph{SIAM J. Optimiz.},
  vol.~19, no.~4, pp. 1574--1609, Jan. 2009.

\bibitem{Duchi_08}
J.~Duchi, S.~Shalev-Shwartz, Y.~Singer, and T.~Chandra, ``Efficient projections
  onto the l1-ball for learning in high dimensions,'' in \emph{Proc.\ ICML},
  Helsinki, Finland, Jul. 2008, pp. 272--279.

\bibitem{WangSimp_13}
W.~Wang and M.~A. Carreira-Perpi\"{n}\'{a}n, ``Projection onto the probability
  simplex: An efficient algorithm with a simple proof, and an application,''
  arXiv:1309.1541, 2013.

\bibitem{Witten_09}
D.~M. Witten, R.~Tibshirani, and T.~Hastie, ``A penalized matrix decomposition,
  with applications to sparse principal components and canonical correlation
  analysis,'' \emph{Biostatistics}, vol.~10, no.~3, pp. 515--534, 2009.

\bibitem{Zou_05}
H.~Zou and T.~Hastie, ``Regularization and variable selection via the elastic
  net,'' \emph{J. Roy. Statist. Soc. Ser. B}, vol.~67, no.~2, pp. 301--320,
  2005.

\bibitem{Sind_12}
V.~Sindhwani and A.~Ghoting, ``Large-scale distributed non-negative sparse
  coding and sparse dictionary learning,'' in \emph{Proc.\ KDD}, Beijing,
  China, Aug. 2012, pp. 489--497.

\bibitem{Duchi_09}
\BIBentryALTinterwordspacing
J.~Duchi, ``Elastic net projections,'' 2009. [Online]. Available:
  \url{http://stanford.edu/~jduchi/projects/proj_elastic_net.pdf}
\BIBentrySTDinterwordspacing

\bibitem{Kim_12}
J.~Kim, R.~Monteiro, and H.~Park, ``Group sparsity in nonnegative matrix
  factorization,'' in \emph{Proc.\ SDM}, Anaheim, California, USA, Apr. 2012,
  pp. 851--862.

\bibitem{Liu_11}
X.~Liu, H.~Lu, and H.~Gu, ``Group sparse non-negative matrix factorization for
  multi-manifold learning,'' in \emph{Proc.\ BMVC}, Dundee, UK, Aug. 2011, pp.
  1--11.

\bibitem{Friedman_10}
J.~Friedman, T.~Hastie, and R.~Tibshirani, ``A note on the group lasso and a
  sparse group lasso,'' arXiv:1001.0736, 2010.

\bibitem{Simon_13}
N.~Simon, J.~Friedman, T.~Hastie, and R.~Tibshirani, ``A sparse-group lasso,''
  \emph{J. Comput. Graph. Statist.}, vol.~22, pp. 231--245, 2013.

\bibitem{Yuan_06}
M.~Yuan and Y.~Lin, ``Model selection and estimation in regression with grouped
  variables,'' \emph{J. Roy. Statist. Soc. Ser. B}, vol.~68, no.~1, pp. 49--67,
  2006.

\bibitem{YaleB_01}
A.~S. Georghiades, P.~N. Belhumeur, and D.~J. Kriegman, ``From few to many:
  illumination cone models for face recognition under variable lighting and
  pose,'' \emph{IEEE Trans. Pattern Anal. Mach. Intell.}, vol.~23, no.~6, pp.
  643--660, Jun. 2001.

\bibitem{i2r_04}
L.~Li, W.~Huang, I.~Y.-H. Gu, and Q.~Tian, ``Statistical modeling of complex
  backgrounds for foreground object detection,'' \emph{IEEE Trans. Image
  Process.}, vol.~13, no.~11, pp. 1459--1472, Nov. 2004.

\bibitem{Tan_13a}
V.~Y.~F. Tan and C.~F\'evotte, ``Automatic relevance determination in
  nonnegative matrix factorization with the $\beta$-divergence,'' \emph{IEEE
  Trans. Pattern Anal. Mach. Intell.}, vol.~35, no.~7, pp. 1592--1605, Jul.
  2013.

\bibitem{Bishop_98}
C.~M. Bishop, ``Bayesian {PCA},'' in \emph{Proc.\ NIPS}, Denver, USA, Jan.
  1999, pp. 382--–388.

\bibitem{Boyd_11}
S.~Boyd, N.~Parikh, E.~Chu, B.~Peleato, and J.~Eckstein, ``Distributed
  optimization and statistical learning via the alternating direction method of
  multipliers,'' \emph{Found. Trends Mach. Learn.}, vol.~3, no.~1, pp. 1--122,
  Jan. 2011.

\bibitem{PSNR_98}
T.~Veldhuizen, ``Measures of image quality,''
  \url{http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/VELDHUIZEN/node18.html},
  1998.

\bibitem{Candes_11}
E.~Cand\`es, X.~Li, Y.~Ma, and J.~Wright, ``Robust principal component
  analysis?'' \emph{J. ACM}, vol.~58, no.~3, pp. 1--37, Jun. 2011.

\bibitem{Netra_14}
P.~Netrapalli, N.~U~N, S.~Sanghavi, A.~Anandkumar, and P.~Jain, ``Non-convex
  robust pca,'' in \emph{Proc.\ NIPS}, Montr\'eal, Canada, Dec. 2014, pp.
  1107--1115.

\bibitem{Fev_09}
C.~F\'evotte, N.~Bertin, and J.-L. Durrieu, ``Nonnegative matrix factorization
  with the {I}takura-{S}aito divergence. {W}ith application to music
  analysis,'' \emph{Neural Comput.}, vol.~21, no.~3, pp. 793--830, Mar. 2009.

\bibitem{Fev_11}
C.~F\'evotte and J.~Idier, ``Algorithms for nonnegative matrix factorization
  with the beta-divergence,'' \emph{Neural Comput.}, vol.~23, no.~9, pp.
  2421--2456, 2011.

\bibitem{Bao_15}
C.~Bao, H.~Ji, Y.~Quan, and Z.~Shen, ``Dictionary learning for sparse coding:
  Algorithms and analysis,'' \emph{IEEE Trans. Pattern Anal. Mach. Intell.},
  vol.~38, no.~7, pp. 1356--1369, July 2015.

\bibitem{Bon_98}
J.~F. Bonnans and A.~Shapiro, ``Optimization problems with perturbations: A
  guided tour,'' \emph{{SIAM Rev.}}, no.~2, pp. 228–--264, 1998.

\bibitem{Syd_05}
K.~Sydsaeter, P.~Hammond, A.~Seierstad, and A.~Strom, \emph{{Further
  Mathematics for Economic Analysis}}.\hskip 1em plus 0.5em minus 0.4em\relax
  Pearson, 2005.

\bibitem{Bill_86}
P.~Billingsley, \emph{Probability and Measure}, 2nd~ed.\hskip 1em plus 0.5em
  minus 0.4em\relax John Wiley \& Sons, 1986.

\bibitem{Fisk_65}
D.~L. Fisk, ``Quasi-martingales,'' \emph{Trans. Amer. Math. Soc.}, vol. 120,
  no.~3, pp. 369--89, 1965.

\end{thebibliography}
