Joint Visual Denoising and Classification using Deep Learning
\IEEEauthorblockN{Gang Chen and Sargur N. Srihari
Visual restoration and recognition are traditionally addressed in pipeline fashion, i.e. denoising followed by classification. Instead, observing correlations between the two tasks, for example clearer image will lead to better categorization and vice visa, we propose a joint framework for visual restoration and recognition for handwritten images, inspired by advances in deep autoencoder and multi-modality learning. Our model is a 3-pathway deep architecture with a hidden-layer representation which is shared by multi-inputs and outputs, and each branch can be composed of a multi-layer deep model. %, and a shared representation can be learned in an discriminative manner to tackle multi-task learning problem. Thus, visual restoration and classification can be unified using shared representation via non-linear mapping, and model parameters can be learnt via backpropagation. Using MNIST and USPS data corrupted with structured noise, the proposed framework performs at least 20\% better in classification than separate pipelines, as well as clearer recovered images. The noise model and the reproducible source code is available at {\url{https://github.com/ganggit/jointmodel.
icip2016_jointlearning.tex
% Sample LaTeX file for creating a paper in the Morgan Kaufmannn two
% column, 8 1/2 by 11 inch proceedings format.


% Template for ICIP-2013 paper; to be used with:
%          spconf.sty  - ICASSP/ICIP LaTeX style file, and
%          IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
\documentclass{article}
\usepackage{spconf,amsmath,graphicx}


\usepackage{subfigure} 
\usepackage{amsmath,amsthm}
\usepackage{slashbox}
\usepackage{multirow}
\usepackage{amssymb}

\newcommand{\argmin}{\arg\!\min}
\newcommand{\argmax}{\operatornamewithlimits{argmax}}

\usepackage{hyperref}


% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}

% Title.
% ------
\title{Joint Visual Denoising and Classification using Deep Learning}

\name{Gang Chen$^{\dagger}$ \qquad Yawei Li$^{\star}$ \qquad Sargur N. Srihari$^{\dagger}$}

\address{$^{\dagger}$ Department of Computer Science, SUNY at Buffalo, Buffalo NY 14260\\
    $^{\star}$ School of Communication and Information Engineering \\
    University of Electronic Science and Technology of China \\Chengdu, Sichuan 611731 China}
%
% Single address.
% ---------------
%\name{Gang Chen and Sargur N. Srihari}
%\address{Department of Computer Science and Engineering\\
%University at Buffalo, The State University of New York\\
%Buffalo, NY 14260\\
%Email: \{gangchen,srihari\}@buffalo.edu}
%
% For example:
% ------------
%\address{School\\
%	Department\\
%	Address}
%
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
%\twoauthors
%  {A. Author-one, B. Author-two\sthanks{Thanks to XYZ agency for funding.}}
%	{Department of Computer Science\\
%SUNY at Buffalo\\
%Buffalo, NY 14260\\
%Email: \{gangchen,srihari\}@buffalo.edu}
%  {C. Author-three, D. Author-four\sthanks{The fourth author performed the work
%	while at ...}}
%	{School of Communication and Information Engineering \\
%	University of Electronic Science and Technology of China\\
%	Chengdu, Sichuan 611731 China \\
%	Email: lywuestc@163.com
%	}
%
\begin{document}
%\ninept
%
\maketitle
%
\begin{abstract}
Visual restoration and recognition are traditionally addressed in pipeline fashion, i.e. denoising followed by classification. Instead, observing correlations between the two tasks, for example clearer image will lead to better categorization and vice visa, we propose a joint framework for visual restoration and recognition for handwritten images, inspired by advances in deep autoencoder and multi-modality learning. Our model is a 3-pathway deep architecture with a hidden-layer representation which is shared by multi-inputs and outputs, and each branch can be composed of a multi-layer deep model. %, and a shared representation can be learned in an discriminative manner to tackle multi-task learning problem. 
Thus, visual restoration and classification can be unified using shared representation via non-linear mapping, and model parameters can be learnt via backpropagation. Using MNIST and USPS data corrupted with structured noise, the proposed framework performs at least 20\% better in classification than separate pipelines, as well as clearer recovered images. The noise model and the reproducible source code is available at {\url{https://github.com/ganggit/jointmodel}}.
\end{abstract}

% -------------------------------------------------------------------------------
\section{Introduction}
Common tasks in computer vision, such as image restoration and recognition, are usually regarded as separate tasks, shown in Fig. \ref{fig:pipeline}. In general, image restoration is an important problem whose purpose is to improve image quality in high-level vision tasks. And there is vast literature, most relying on unsupervised approaches, such as Wiener filter \cite{Wiener64}, Markov random field \cite{besag86}, sparse coding \cite{Elad06}, deep learning \cite{Xie12} with regularization terms or prior information of the underlying image. Visual recognition as a supervised task, has been extensively studied in machine learning and computer vision \cite{LeCun89,Larochelle12}. These two problems are characterized by very distinct statistical properties which make it difficult to address them together. Although they come from different input channels, there are connections between these two tasks: (1) the noisy image is derived from its clean one, (2) better image quality will improve recognition tasks. That is also the reason that we need preprocessing stage in many recognition problems. Hence, it is possible to learn useful representations which can potentially be used for such data to handle these two problems together. %gang14

Recent advances in deep learning \cite{Hinton06b} and multi-modality learning \cite{Ngiam10} shed light on joint representation learning which captures the real-world concept that the data corresponds to. %The basic idea behind multi-modality is to learn a joint deep architecture by combining the modalities . For example, we would like a probabilistic model to 
Deep learning \cite{Hinton06a,Bengio12} can learn abstract and expressive representations, which can capture a huge number of possible input configurations. %Hence, the representation learnt is useful for classification and information retrieval. 
The multimodal learning model \cite{Srivastava14b} in a sense extends the deep learning framework to handle different modalities. Thus it can learn a joint representation such that similarity in the code space indicates similarity of the corresponding concepts. %It is also desirable if the joint representation is robust enough even when the data is typically very noisy and there may be missing values. 
However, these previous multi-modality models \cite{Ngiam10,Srivastava14b} can only handle one task. %, refer to Fig. \ref{fig:dbn3ways}(a). 
Moreover, how to jointly restore and classify images is also a challenge when the data is typically very noisy, e.g. structural noise.

\begin{figure}[t!]
\centering
\includegraphics[trim = 20mm 30mm 30mm 30mm, clip, width=8.5cm]{pipeline2.pdf}
\caption{Models for restoration/classification. (a) restoration; (b) classification; (c) pipeline; (d) joint restoration and recognition.}
\label{fig:pipeline}
\end{figure}

We propose here a unified framework, which can learn a joint model to handle visual restoration and recognition together, refer to Fig. \ref{fig:pipeline}(d). Our one fan-in and two fan-out deep model is a network of 3 different kind of inputs coupled stochastic binary hidden units in a hierarchical structure. The inputs can be binary or real values, and they share a hidden layer via multi-layers non-linear mapping for each input. %An example with a 2-layer deep structure is shown in Fig. \ref{fig:dbn3ways}(b). %In particular, our model is a 3-fan deep model, which has one hidden layer shared by different inputs and outputs, refer to Fig. \ref{fig:dbn3ways}(2) for visual understanding. 
%Therefore, our model is kind of mixture of deep autoencoder for reconstruction and feedforward neural network for classification. Thus, it is more powerful than 2-pathway multi-modal models to handle multi-tasks effectively. In addition, we take a different approach from conventional visual restoration tasks. 
Specifically, visual restoration in our work is supervised, where the hidden and nonlinear structural information is learnt from data, and can handle more complex situations, such as structural denoising or super resolution. Furthermore, classification depends on the shared representation which is correlated to both clear and corrupted inputs. 
%In the learning stage, all inputs are observed, so that we can leverage them to learn a joint representation in a supervised manner. We pretrain the model with contrastive divergence, followed with gradient descent to update model parameters. %For example, the classification prediction depends both on other two branches. In the inference stage, our model only assume corrupted visual input, and we need to restore it and predict its labels or tags simutanously. 
We pretrain the model with contrastive divergence, followed with gradient descent (L-BFGS) to update model parameters. We test our model on character denoising (to remove structured noises) and recognition tasks, and show the advantages over other separate baselines. 
%\begin{figure}[t!]
%\centering
%\includegraphics[trim = 70mm 85mm 45mm 60mm, clip, width=9cm]{dbn_3ways.pdf}
%\caption{(a) an example of a 2-layer deep autoencoder with tied weights. This graph depicts the encoder and decoder, which can be learned with pretraining (RBM) and fine-tuning (backpropagation) process. (b) The right graph shows our unified framework for visual restoration and recognition, with a joint representation shared by different inputs.}
%\label{fig:dbn3ways}
%\end{figure}

% ---------------------------------------------------------------------------------
\section{Related work}
There is little work, which models visual restoration and recognition in a joint framework with deep learning. However, there is much literature either addresses one or the other. The visual restoration problem, especially image denoising and super resolution, focuses on improve the image quality and numerous denoising methods have been proposed \cite{Wiener64,besag86,RudinO94,Tomasi98,Portilla03,Elad06,Jain08,ChenXC12}. %,Chen12}.  
Recent advances in deep learning \cite{Hinton06a,Hinton06b} has attracted great attention in machine learning community, and has been used for visual restoration. For example, %Gallinari et al. proposed an autoassociative memory \cite{Gallinari87}, similar to Hopfield Networks \cite{Hopfield88} for denoising problems. Another insightful work proposed by Seung \cite{Seung98} is to leverage the recurrent neural network for denoising, which advocates the corrupted pattern completion task using backpropagation through time. Recently, 
deep neural networks have been used for denoising and inpainting \cite{Xie12} and yield promising results. The deep denoising autoencoder \cite{Gang14b} extends the work \cite{Hinton06b,Vincent10} for image denoising by minimizing the reconstruction loss to recover the original image. %In their settings, only the clean data is provided while the noisy version of it is generated during training by adding different noises to the clean input, such as additive isotropic Gaussian noise, Salt-and-pepper noise and a zero-masking corruption noise. The stacked denoising autoencoer (SDA) basically passed on the the output of the current layer as the clean data to learn the next layer denoising autoencoder and the procedure repeats. 
Deep learning has also been used for classification tasks, such as character recognition \cite{LeCun89,Hinton06b}, document classification \cite{Larochelle08} and image recognition \cite{Krizhevsky12}. The basic idea \cite{Bengio12} is to leverage deep neural networks, such as deep autoencoder, convolutional neural network \cite{LeCun89} or deep Boltzmann machines \cite{Salakhutdinov12} to learn representations helpful for classification. Recently, multi-modality learning, which generalizes deep learning to handle different input channels, has attracted great attention. For example, \cite{Srivastava14b} leverages deep Boltzmann machines to bridge images and texts. The model is a 2-fan (image and text as input or output) deep structure and learns a shared representation for the two modalities. Similarly, \cite{Ngiam10} et al. leverages deep autoencoder for multimodal deep learning, to handle video and audio data. 
%One interesting work related to our model is to use deep networks for robust visual recognition \cite{Tang10}. Basically, it leverages deep belief networks (DBN) and sparse coding, as well as a probabilistic denoising algorithm to determine a subset of the hidden layer nodes to improve the robustness of recognition tasks. 
Recently, a robust Boltzmann machine (RoBM) \cite{Tang12} was introduced for recognition and denoising. This model added another shape RBM to the Gaussian RBM prior to model the noisy variables which indicate where to ignore the occluder in the image. %Thus, the RoBM contains a multiplicative gating mechanism to handle unexpected corruptions of the observed variables. 
However, the experiments only show its effectiveness for regular structured noise. In this paper, we propose a unified framework, which can learn a joint model to handle visual restoration and recognition together. Our model is a 3-fan deep architecture, which generalizes previous multimodality models \cite{Ngiam10,Srivastava14b,ChenS15} for more complex multi-tasks, such as joint visual denoising and classification. %Moreover, our model is kind of mixture of deep autoencoder and deep Boltzmann machines, which are unified in the joint framework for both reconstruction and classification tasks. 
%Our deep denoising autoencoder (DDAE) extends the deep autoencoder \cite{Hinton06b} for image denoising problem. Unlike deep autoencoder, which requires the clean input, the DDAE requires the corrupted input and its clean output as training pairs to learn model parameters. In the prediction stage, given any noisy input, it will output its corresponding clean data. To enhance the output, we also add feedback loop in the construction stage.  

% --------------------------------------------------------------------------------
\section{Joint model}
Our model for joint visual restoration and recognition is a 3-pathway deep architecture, with restricted Boltzmann machines (RBMs) as the building blocks. From another perspective, our model can be thought as the mixture of deep autoencoder and feedforward network. %In the following parts, we will review RBM first, and then we will introduce our model.
% -------------------------------------------------------------------------------
%\subsection{Background}
%An RBM with $n$ hidden units is a parametric model of the joint distribution between a layer of hidden variables ${\bf h} \in \{0, 1\}^n$ and the observation ${\bf v} \in \{{0,1}\}^D$. The RBM joint likelihood takes the form:
%\begin{equation}\label{eq:eq1}
%p({\bf v},{\bf h}) \propto e^{-E({\bf v}, {\bf h})}
%\end{equation}
%where the energy function is 
%\begin{equation}\label{eq:eq2}
%E({\bf v},{\bf h}) = -{\bf h}^T{\bf W}^1{\bf v} - {\bf b}^T {\bf v} - {\bf c}^T{\bf h} 
%\end{equation}
%% with parameters $\Theta_{1} = \{{\bf W}^1, {\bf b}, {\bf c}\}$. 
%And we can compute the following conditional likelihood:
%\begin{subequations}\label{eq:eq3}
%\begin{align}
%        & p({\bf v} | {\bf h}) = \prod_{i} p( v_{i} | {\bf h})\\
%        & p(v_i=1 | {\bf h}) = \mathrm{logistic}(b_{i} + \sum_{j} W^1(i,j) h_j)\\
%        & p(h_i =1 | {\bf v}) = \mathrm{logistic}(c_{i} + \sum_{j} W^1(j, i) v_j)
%\end{align}
%\end{subequations}
%where $\mathrm{logistic}(x)  = 1/(1+e^{-x})$. To learn RBM parameters, we need to minimize the negative log likelihood $-\mathrm{log}p({\bf v})$ on training data, the parameters updating can be calculated with an efficient stochastic descent method, namely contrastive divergence (CD) \cite{Hinton06a}.% to the negative log likelihood.
%%$\{{\bf W}_{i}\}_{i=1}^L$

%The traditional Autoencoder (AE) \cite{Hinton06b} relies on the deterministic mapping function $f_\theta$ that transforms an input vector ${\bf v} \in \mathbb{R}^D$ into hidden representation ${\bf h}$ with $n$ neurons, whose typical form is an affine transformation followed by a linear or nonlinear mapping
%\begin{equation}\label{eq:encoder}
% f_\theta({\bf v}) = s({\bf Wv+b})
%\end{equation}
%where the parameter $\theta = \{{\bf W}, {\bf b}\}$, with ${\bf W} \in \mathbb{R}^{n\times D}$ and ${\bf b} \in  \mathbb{R}^{n}$. The resulting hidden representation {\bf h} is then mapped back to reconstruct the input in the original space ${\bf \hat{v}} = g_{\theta^\prime}({\bf h})$. More specifically, the mapping function is
%\begin{equation}\label{eq:decoder}
% g_{\theta^\prime} ({\bf h}) = s({\bf W^\prime h+b^\prime})
%\end{equation}
%with $\theta^\prime=\{ {\bf W^\prime}, {\bf b^\prime} \}$. The function $s$ in Eqs. (\ref{eq:encoder}) and (\ref{eq:decoder}) can be linear or nonlinear mapping, such as logistic function. If it is a tied weight autoencoder, then $\theta^\prime = \theta^T$. AE can be trained by minimizing the reconstruction error or cross entropy with various optimization methods \cite{Hinton06b,Vincent10,Gang14b}, %For example, given the original data ${\bf v}_{i}$, it can minimize the following reconstruction error:
%\begin{equation}\label{eq:obj}
%{\theta, \theta^\prime} = \textrm{argmin}_{\substack{\theta, \theta^\prime}} \sum_{i = 1}^N ({\bf v}_i - {\bf \hat{v}}_i)^2 = \textrm{argmin}_{\theta, \theta^\prime} \sum_{i = 1}^N ({\bf v}_i - g\circ f({\bf v}_i))^2
%\end{equation}
%where we ignore the underscript parameters in the mapping functions $f$ and $g$ for convenience, and $\circ$ indicates function composition.

%A denoising autoencoder (DAE) generalizes the AE with the clean and corrupted pairs $ \langle {\bf v}, {\bf \tilde{v}} \rangle$, and it can be trained %with the clean and its corrupted pairs, and then it can be used to 
%to reconstruct a clean ``repaired" input from a corrupted version of it. %Given the original data ${\bf v}_{i}$ and its corrupted ${\bf \tilde{v}}_{i}$ for $i = \{1,2,...,N\}$, it minimizes the following reconstruction error:
% The key difference between AE and DAE is that $f$ is now a deterministic function of ${\bf \tilde{v}}$ rather than ${\bf v}$. %where ${\bf \tilde{v}}$ is the corrupted version of the clean ${\bf v}$. %In order to learn an denoising autoencoder, we need the clean ${\bf v}$ and the corresponding corrupted ${\bf \tilde{v}}$. %Thus, the DAE can learn a far more clever mapping than AE: one that extracts features useful for denoising.
%Note that the purpose of the denoising auto-encoder is to learn more robust features to handle the random noise setting. %Hence, DAE can predict the missing values from the uncorrupted values and it can generally handle the random noise setting. 


%Secondly, unlike other models with deep architectures, the approximate inference procedure for DBMs incorporates a top-down feedback in addition to the usual bottom-up pass, allowing Deep Boltzmann Machines to better incorporate uncertainty about missing or noisy inputs.

%After the pretraining for the encoder, the RBMs are ``unrolled" to create a deep autoencoder by initially mirroring the weights between the encoder and the decoder,
%%After pretraining multiple layers of feature detectors, the model to produce encoder and decoder networks that initially use the same weights,
%refer to an example DDAE with 2-layers in Fig. \ref{fig:dae}. Then, we minimize the reconstruction error in Eq. (\ref{eq:deepobj}) with the global fine-tuning stage, which %replaces stochastic activities by deterministic, real-valued probabilities and 
%uses backpropagation through the whole deep denoising auto-encoder to fine-tune the weights for optimal reconstruction. The difference between our DDAE and the deep autoencoder \cite{Hinton06b} is that DDAE learns a mapping from a noisy observation to the clean of it, while the deep autoencoder tries to find a projection to reconstruct the original data.

%Again, all units were logistic except for the 30 linear units in the code layer. After fine-tuning on all 60,000 training images, the autoencoder was tested on 10,000 new images and produced much better reconstructions than did PCA (Fig. 2B). A two-dimensional autoencoder produced a better visualization of the data than did the first two principal components (Fig. 3).

% -------------------------------------------------------------------------------------
\subsection{Objective function}
In this part, we will present a jointly learning model for visual restoration and recognition. Assume that we have a training set $\mathcal{D} = \langle {\bf \tilde{v}}_{i},{\bf v}_i, {\bf y}_i \rangle_{i=1}^N$, with the corrupted image ${\bf \tilde{v}}_{i} \in \mathbb{R}_{+}^D$, the clear image ${\bf v}_{i} \in \mathbb{R}_{+}^D$ and its corresponding label ${\bf y}_{i} \in \{0,1\}^K$, for $i = \{1,2,...,N\}$. The purpose of our model is to learn a shared hidden representation in the deep architecture, which can restore the original image and label it given the noisy input. %The model structure is shown in Fig. \ref{fig:dbn3ways}(b). 
Thus, given the training triplet $\langle {\bf \tilde{v}}_{i}, {\bf v}_{i}, {\bf y}_{i} \rangle$, we use the following cross entropy loss:
%\begin{align}\label{eq:deepobj}
%& \{{\theta_{i}\}_{i=1}^L, \{\theta^\prime_i\}_{i = 1}^L} = \nonumber \\
%& \textrm{argmin}_{\theta, \theta^\prime} \sum_{i = 1}^N \big({\bf v}_i - \underbrace{g_1 \circ g_2\circ \cdot\cdot\cdot \circ g_{L}}_{L} \circ \underbrace{f_{L} \circ f_{L-1} \circ \cdot\cdot\cdot  \circ f_1}_{L}({\bf \tilde{v}}_i)  \big)^2
%\end{align}
\begin{align}%\label{eq:deepobj}
& \{{\theta_{i}\}_{i=1}^L, \{\theta^\prime_i\}_{i = 1}^L}, \{\theta^{\prime\prime}_i\}_{i = 1}^L  =\textrm{argmin}_{\theta, \theta^\prime, \theta^{\prime\prime}} \mathcal{L}({\bf \tilde{v}}_{i}, {\bf v}_{i}, {\bf y}_{i}; \theta, \theta^\prime, \theta^{\prime\prime})  \nonumber \\
&= \textrm{argmin}_{\theta, \theta^\prime, \theta^{\prime\prime}}   -  \sum_{i = 1}^N  {\bf v}_i \textrm{log} {\bf \hat{v}}_i +  (1- {\bf v}_i) \textrm{log} (1-{\bf \hat{v}}_i)  \nonumber \\
&  - \lambda \sum_{i = 1}^N  {\bf y}_i \textrm{log} {\bf \hat{y}}_i +  (1- {\bf y}_i) \textrm{log} (1-{\bf \hat{y}}_i) \label{eq:deepobj}
%& \textrm{with } {\bf \hat{v}}_i  =   \underbrace{g_1 \circ g_2\circ \cdot\cdot\cdot \circ g_{L}}_{L\textrm{ times}} \circ \underbrace{f_{L} \circ f_{L-1} \circ \cdot\cdot\cdot  \circ f_1}_{L \textrm{ times}}({\bf \tilde{v}}_i) \label{eq:eqb}
\end{align}
where $\{\theta, \theta^\prime, \theta^{\prime\prime} \}$ are the weights in the 3-way deep architecture respectively (we ignore the subscripts for clarity), $\lambda$ is the weight to balance the two losses. And ${\bf \hat{v}}_i$ and ${\bf \hat{y}}_i$ are the prediction from the noise input ${\bf \tilde{v}}_{i}$, specified as follows
\begin{align} 
& {{\bf h}_i} = \underbrace{f_{L} \circ f_{L-1} \circ \cdot\cdot\cdot  \circ f_1}_{L \textrm{ times}}({\bf \tilde{v}}_i) \label{eq:eqhidden} \\
 & {\bf \hat{v}}_i  =   \underbrace{g_1 \circ g_2\circ \cdot\cdot\cdot \circ g_{L}}_{L\textrm{ times}}  ({{\bf h}_i}) \label{eq:eqpred} \\
 & {\bf \hat{y}}_i  =   \underbrace{\phi_1 \circ \phi_2\circ \cdot\cdot\cdot \circ \phi_{L}}_{L\textrm{ times}}  ({{\bf h}_i}) \label{eq:eqlabels}
\end{align}
where $f_l$, $g_l$, and $\phi_l$ are non-linear projection functions, with weight parameters $\theta_l$, $\theta_l^\prime$ and $\theta_l^{\prime\prime}$ respectively in each layer. We ignore the underscript for parameters in mapping functions $f_l$, $g_l$, and $\phi_l$, for $l = \{1, ..., L\}$ in the above equations. As the deep belief network (DBN), we use the same logistic (or sigmoid) function in each layer. In our model, we attempt to learn the top layer hidden representations ${\bf h}_i$ via Eq. \ref{eq:eqhidden}, which are shared by the triplet $\langle {\bf \tilde{v}}_{i}, {\bf v}_{i}, {\bf y}_{i} \rangle$. In the predication stage, we hope to restore the clear image ${\bf v}_{i}$ and its label ${\bf y}_{i}$, using function compositions $\{g_l\}_{l=1}^L$ and $\{\phi_l\}_{l=1}^L$ respectively from the hidden layer ${\bf h}_i$.  Thus, we can think Eq. \ref{eq:eqhidden} as the encoding step, while Eqs. \ref{eq:eqpred} and \ref{eq:eqlabels} are the decoding steps. Our model has three pathways with the shared top layer $L$ in the center. %An example of our model with $L=2$ is shown in Fig. \ref{fig:dbn3ways}(b). Note that we use the same number of layers for each branch in the deep architecture only for clarity concern. 
In practice, we can select different number of layers and hidden nodes to predict ${\bf v}_{i}$ and ${\bf y}_{i}$ respectively. %Note that ${\bf \tilde{v}}_{i}$ and ${\bf v}_{i}$ can be set with different dimensionality in the super-resolution tasks.

\subsection{ Learning and Inference}
Our deep model is a 3-branch deep structure with shared representations, which is different from the deep autoencoder. Thus, it is more complex to learn model parameters. In general, the parameters in each pathway can be pretrained separately in a completely unsupervised fashion, which allows us to leverage a large supply of unlabeled data. And the complexity of the pretraining depends on the number of layers and nodes in each pathway. %In practice, we first use pretraining to initialize the weights for each branch using DBN. Basically, it is to learn a stack of RBMs greedily layer-by-layer. %To put it simply, the learned features of the current layer RBM are treated as the ``data" to train the next RBM in the stack. 
For each pathway, we can initialize the weight parameters in each layer with DBN simultaneously. Then, we can infer the top hidden layer with mean field methods \cite{Salakhutdinov12} and update the top layer weights for each branch with constrative divergence (CD) \cite{Hinton06a}. After pretraining, we minimize the reconstruction error in Eq. (\ref{eq:deepobj}) with the global fine-tuning stage, which %replaces stochastic activities by deterministic, real-valued probabilities and 
uses backpropagation through the whole network to compute gradients and then fine-tune the weights for optimal reconstruction and recognition (local minimum). Note that the gradients w.r.t. the shared hidden representations should be the summation from the two cross entropy losses in Eq. (\ref{eq:deepobj}). 

In the inference stage, for a corrupted image ${\bf \tilde{v}}$, we first use Eq. \ref{eq:eqhidden} to project it into the hidden space, and then reconstruct the clear ${\bf v}$ and predict its label ${\bf y}$.
%The difference between our DDAE and the deep autoencoder \cite{Hinton06b} is that DDAE learns a mapping from a noisy observation to the clean of it, while the deep autoencoder tries to find a projection to reconstruct the original data.
%{\bf Learning}
%To learn the parameters in Eq. (\ref{eq:deepobj}).
%{\bf Inference}
%Note that the deep autoencoder can learn hidden representations to reconstruct its self. Compared to the deep autoencoder, our model has two outputs (while the deep autoencoder only has one), one branch for restoration or denoising, and the other branch for recognition. 
%The deep denoising autoencoder (DDAE) here is similar to the deep autoencoder, both of which are consisted of an encoder and a symmetric decoder \cite{Hinton06b}. The difference between our model and the deep autoencoder is that our model needs noisy and clean data pairs to learn a neural network for denoising problems, while the deep autoencoder requires the input and the output are the same for representation learning.


%\subsection{Relationship to deep autoencoder}
%%A Multimodal DBM can be viewed as a composition of unimodal undirected pathways. Each path- way can be pretrained separately in a completely unsupervised fashion, which allows us to leverage a large supply of unlabeled data. Any number of pathways each with any number of layers could potentially be used. The type of the lower-level RBMs in each pathway could be different, account- ing for different input distributions, as long as the final hidden representations at the end of each pathway are of the same type.
%The left branch over the shared hidden layer in  Fig. \ref{fig:dbn3ways}(2) from the input ${\bf \tilde{v}}$ to output ${\bf v}$ generalizes deep autoencoder proposed by Hinton \cite{Hinton06b}. In our framework, we learn a nonlinear mapping from ${\bf \tilde{v}}$ to output ${\bf v}$, and simultaneously we also learn a classifier on the right branch for recognition task. Note that the deep autoencoder proposed by Hinton \cite{Hinton06b} is an unsupervised method to learn a hidden representation with deep neural network to reconstruct the input, which cannot be used for denoising or super resolution. On the contrary, our model is totally a supervised approach, by learning the shared representations for both restoration and recognition. %Unlike stacked denoising autoencoder (SDA) to form a deep network by using hidden layer activation of the previous layer as input of the next layer for feature learning, our model is a global supervised method for image denoising, which can be trained to reconstruct a ``clean" input from a noisy version of it. 
% 
% Our model is related to multimodal learning with the goal to learn a final representation which are shared by different input channels. However, the multi-modality learning needs to maximize the joint probability over observations, and its inference requires approximate techniques, such as variational Bayes \cite{Srivastava14b}. Our model is builded upon the same principle, the shared hidden layer among different inputs, but we minimize the cross entropy losses, which can be optimized under gradient-based methods, such as L-BFGS. In a sense, our model is more related to multi-task learning. 
%%\subsubsection{Modeling multitasks}
% --------------------------------------------------------------------------------------
\section{Experiments}
We analyzed our model on handwriting denoising problems on several standard handwriting datasets. We evaluated the denoising performance with Peak signal-to-noise ratio (PSNR) and recognition tasks with error rate.% to evaluate whether our joint model is helpful or not over the separate pipeline in the recognition tasks. 
%We will then show that denoising with the DDAE is significantly better than standard algorithms on the MNIST and USPS alphabets. Finally, we investigate the recognition performance when test images contain structural noise or occlusions on the MNIST and USPS Database.

\subsection{Data description}
%\noindent
{\bf The MNIST dataset}\footnote{\url{http://yann.lecun.com/exdb/mnist/}} consists of $28\times28$-size images of handwriting digits from $0$ through $9$ with a training set of 60,000 examples and a testing set of 10,000 examples, and has been widely used to test character denoising and recognition methods. A set of examples are shown in Fig. (\ref{fig:mnist_regular}). \\%In the experiment, we test both denoising and recognition performance.
%\noindent
{\bf The USPS Handwritten binary Alphadigits}\footnote{\url{http://www.cs.nyu.edu/~roweis/data/binaryalphadigs.mat}} are binary images with size $20 \times16$. There are digits of ``0" through ``9" and capital ``A" through ``Z", with 39 examples of each class. In our experiments, we only test our method on the binary Alphabets.

\subsection{Experimental setting}
In all experiments, we first use the stacked RBMs to initialize the model weights for all layers, and $\lambda=1$ to balance the two losses. In the fine-tuning state, we use the L-BFGS to optimize the model parameters. For the MNIST digits, we set the number of hidden nodes (encoding) [400 200 250 100] in the 4-layer deep model, the restoration output mirrors the setting except the last layer set as the same dimension as the input, and the recognition output has the same setting except the last layer set as the number of classes. For the USPS alphabets, we use the two layer deep encoding structure, with hidden nodes 100 and 64 respectively in each layer in the experiments. \\%We do not try other number of layers or nodes in each layer. 
{\bf Noise model}
We consider two kinds of structured noise that are widely appeared in the handwriting images. \\
{(1) The type 1 noise}: horizontal/vertical lines and sine waves, refer to Fig. \ref{fig:mnist_regular}(a) for visual understanding. \\
{(2) The type 2 noise}: random lines/strokes, refer the structural noise in Fig. \ref{fig:denoised}(b). 
Basically, the type 1 noise could corrupt images lightly, while the type 2 noise would heavily corrupt images, with more than 50\% regions. 
\\
{\bf Baselines}
We compare our method to Wiener \cite{Wiener64}, RoBM \cite{Tang12} and deep denoising autoencoder (DDAE) \cite{Gang14b}. %For these baselines, we first used these denoising methods to restore the original images. Then, we trained the deep neural network (DNN) \cite{Hinton06b} with the clear images and its corresponding labels. And then, we used the DNN model on the reconstructed images to evaluate their performance. 
\subsection{Results}
%
We first consider to remove the type 1 structural noise in the handwriting images. % refer to Fig. \ref{fig:mnist_regular}(b) for visual understanding. 
To generate the clean and noisy pairs, we add the type 1 noise to each MNIST image by randomly sampling horizontal/vertical lines or sine waves to construct its noisy observation. Then, we train our joint model on the 60,000 triplets (the clean, its noisy image and corresponding label), and test on the 10,000 noisy testing dataset for restoration and recognition. Analogously, we take the same way to train our model with type 2 noise.

For the baselines, we first learn the deep neural network (DNN) \cite{Hinton06b} on the clean MNIST 60,000 images for classification, %tasks by minimizing the cross entropy. In the experiment on the MNIST digits, we used the 
with default parameters, namely 4 layers with hidden nodes [500 500 2000 10] respectively for each layer. %Then we trained the model on the clean MNIST 60,000 images to learn the DNN model. 
The error rate on the clean testing set we can get using DNN is 1.2\%, while the error rates on the noisy testing set are 41.9\% and 61.0\% respectively with the lightly and heavily corrupted noise in Figs. \ref{fig:mnist_regular}(a) and \ref{fig:denoised}(b). Then, we use the model learned to test the denoising baselines on the recognition task. 

The sampled results with our model are shown in Fig. \ref{fig:mnist_regular} (b), while the quantitative results were shown in Table (\ref{tab:mnistwave}). The lower bound of PSNR for the type 1 noise is $9.7$ dB, which is calculated on the noisy testing set. From the denoised results, we can see that our model is superior to the competitive baselines on both denoising and recognition. In other words, our joint model by leveraging label information for visual restoration is significant better than separate pipelines. We also test our method on the type 2 noise, refer noisy examples in Fig. \ref{fig:denoised}(b) and its denoised ones in Fig. \ref{fig:denoised}(d), as well as the quantitative performance in Table (\ref{tab:mnist}). %The lower bound of the PSNR for the type 2 noise is $7.65$ dB. We also evaluate the quantitative performance in Table (\ref{tab:mnist}) and it demonstrates that our methods is better than other method on removal of the structural noise and recognition tasks. 

Apart from the digits, we also test our method on the USPS alphabets with the type 2 noise. Similar to the experiment on the MNIST digits, we add random strokes to the alphabets to create the noisy observations. Because there are only 39 training images for each class, we generated 10 corrupted samples for each clean image. Then we divided the clean and noisy binary pairs into the training set (account for 80\%) and testing set (the rest 20\%). We trained our joint model on the training set and test its performance on the testing set. The visual performance of our approach is shown in Fig. \ref{fig:usps} (d). The quantitative comparison between our method and the baselines is shown in Table (\ref{tab:usps}), which demonstrates that our method yields better denoising and labeling results. 
%It demonstrates that our joint model can boost both denoising and recognition performance. 
\begin{figure}[t!]
\centering
\begin{tabular}{cc}
%\includegraphics[trim = 48mm 99mm 45mm 95mm, clip, width=4.4cm]{mnist_noise1_clear.pdf}&
\includegraphics[trim = 48mm 99mm 45mm 95mm, clip, width=4.2cm]{mnist_ddae_noise1_corrupted.pdf} &
\includegraphics[trim = 48mm 99mm 45mm 95mm, clip, width=4.2cm]{mnist_noise1_rec.pdf} \\
(a) & (b) %\\
%\includegraphics[trim = 48mm 99mm 45mm 95mm, clip, width=4.4cm]{mnist_ddae_noise1_rec.pdf} &
%\includegraphics[trim = 48mm 99mm 45mm 95mm, clip, width=4.4cm]{mnist_noise1_rec.pdf} \\
%(c) & (d)
\end{tabular}
\caption{The denoising results comparison on the lightly corrupted MNIST dataset (the type 1 noise). (a) noisy images with horizontal/vertical lines/sine waves; (b) denoising results with our joint model. }%It demonstrates that DDAE works well on the structural noise. It also shows that it is useful to add feedback into DDAE to enhance the results.}
\label{fig:mnist_regular}
\end{figure}

\begin{table}[t!]
\centering
\resizebox{.9\columnwidth}{!}{
\begin{tabular}{lrr}
\hline
Model & PSNR (dB) & Error rate (\%) \\
\hline
Wiener \cite{Wiener64} & 13.7 & 22.4\\% DRBM ($\eta = 0.0005$, $n = 50$) & 27.6\\
RoBM \cite{Tang12} & 15.8 & 17.2\\
DDAE w/o loop \cite{Gang14b} & 16.1 &  7.10 \\
DDAE with loop \cite{Gang14b}  & 17.7 &  4.36\\
Our method & 19.64 &  3.75 \\
\hline
% DNN (noisy data) & NA & 41.9\\
DNN \cite{Hinton06b} & $\geq 9.7$ & $1.20 \sim 41.9$\\
\hline
%  20news & 30.8 & - & 30.6 & 0.026 & 32.2 & 24.9 & 23.9\\
\end{tabular}}
\caption{The experimental comparison on the lightly corrupted MNIST digits (type 1 noise). The PSNR value on the noisy testing set is 9.7 dB, which can be thought as the lower bound. The error rate range using DNN means that the error rates on the noisy testing set and the original clean testing set are 41.9\% and 1.2\% respectively. It demonstrates that our joint model can boost both denoising and recognition performance.}
\label{tab:mnistwave}
\end{table}
%\begin{figure*}[t!]
%\centering
%\begin{tabular}{cccc}
%\includegraphics[trim = 48mm 95mm 45mm 95mm, clip, width=4.2cm]{mnist_orig_regular.pdf}&
%\includegraphics[trim = 48mm 95mm 45mm 95mm, clip, width=4.2cm]{mnist_noise_regular.pdf} &
%\includegraphics[trim = 48mm 95mm 45mm 95mm, clip, width=4.2cm]{mnist_noloop_regular.pdf} &
%\includegraphics[trim = 48mm 95mm 45mm 95mm, clip, width=4.2cm]{mnist_loop_regular.pdf} \\
%(a) & (b) & (c) & (d)
%\end{tabular}
%\caption{The denoising results comparison on the MNIST dataset. (a) original images; (b) noisy images with random structures; (c) denoising results with deep denoising autoencoder without feedback loop; (d) denoising results with enhanced deep denoising autoencoder (with feedback loop). It demonstrates that DDAE works well on the structural noise. It also shows that it is useful to add feedback into DDAE to enhance the results.}
%\label{fig:denoised}
%\end{figure*}
\begin{figure}[h!]
\centering
\begin{tabular}{cc}
\includegraphics[trim = 48mm 99mm 45mm 95mm, clip, width=4.2cm]{mnist_noise2_org2_4layers.pdf}&
\includegraphics[trim = 48mm 99mm 45mm 95mm, clip, width=4.2cm]{mnist_ddae_noise2_corrupted.pdf} \\
(a) & (b) \\
\includegraphics[trim = 48mm 99mm 45mm 95mm, clip, width=4.2cm]{mnist_ddae_noise2_rec.pdf} &
\includegraphics[trim = 48mm 99mm 45mm 95mm, clip, width=4.2cm]{mnist_noise2_rec2_4layers.pdf} \\ % % mnist_noise2_rec.pdf
(c) & (d)
\end{tabular}
\caption{The denoising results comparison on the heavily occluded MNIST digits (the type 2 noise). (a) original images; (b) noisy images with random structures; (c) denoising results with deep denoising autoencoder; (d) denoising results with our joint model. }%It demonstrates our model can effectively remove structural noises.}%It demonstrates that DDAE works well on the structural noise. It also shows that it is useful to add feedback into DDAE to enhance the results.}
\label{fig:denoised}
\end{figure}

\begin{table}[t!]
\centering
\resizebox{.85\columnwidth}{!}{
\begin{tabular}{lrr}
\hline
Model & PSNR (dB) & Error rate (\%) \\
\hline
Wiener \cite{Wiener64} & 11.7 & 58.5\\% DRBM ($\eta = 0.0005$, $n = 50$) & 27.6\\
RoBM \cite{Tang12} & 13.9 & 52.6\\
DDAE w/o loop \cite{Gang14b} & 13.58 &  35.9 \\
DDAE with loop \cite{Gang14b} & 15.15 &  29.9\\
Our method & 18.6 &  12.7\\
\hline
DNN \cite{Hinton06b}& $\geq7.65$ & $1.20 \sim 61.0$\\
\hline
%  20news & 30.8 & - & 30.6 & 0.026 & 32.2 & 24.9 & 23.9\\
\end{tabular}}
\caption{The experimental comparison on the heavily corrupted MNIST digits (type 2 noise).} %It indicates that our method is better than competitive baselines on both denoising and recognition tasks.}
\label{tab:mnist}
\end{table}

%\begin{table}[h!]
%\centering
%\resizebox{.6\columnwidth}{!}{%
%\begin{tabular}{lr}
%\hline
%Model & PSNR  \\
%\hline
%Wiener \cite{Wiener64} & 13.7 \\% DRBM ($\eta = 0.0005$, $n = 50$) & 27.6\\
%ROBM \cite{Tang12} & 13.9 \\
%DDAE w/o loop & 13.58 \\
%DDAE with loop & 15.15 \\
%\hline
%%  20news & 30.8 & - & 30.6 & 0.026 & 32.2 & 24.9 & 23.9\\
%\end{tabular}}
%\caption{The experimental comparison on the MNIST dataset. }
%\label{tab:mnist}
%\end{table}
\begin{table}[t!]
\centering
\resizebox{.85\columnwidth}{!}{
\begin{tabular}{lrr}
\hline
Model & PSNR (dB) & Error rate (\%) \\
\hline
Wiener \cite{Wiener64} & 14.2 & 67.8 \\% DRBM ($\eta = 0.0005$, $n = 50$) & 27.6\\
RoBM \cite{Tang12} & 16.3 & 62.8\\
DDAE w/o loop \cite{Gang14b} & 19.2&  42.5 \\
DDAE with loop \cite{Gang14b} & 18.5 &  44.1\\
Our method & 19.6 &  32.8\\
\hline
DNN \cite{Hinton06b}& $\geq 8.12$ & $1.29 \sim 67.4$\\
\hline 
%  20news & 30.8 & - & 30.6 & 0.026 & 32.2 & 24.9 & 23.9\\
\end{tabular}}
\caption{The experimental comparison on USPS alphabets (type 2 noise). The PSNR value of DNN is 8.12 dB, which shows the lower bound on the noisy testing set. The error rate range using DNN means that the error rates on the noisy testing set and the original clean testing set are 67.4\% and 1.29\% respectively.} %It demonstrates that our method is better than competitive baselines on both denoising and recognition tasks.}
\label{tab:usps}
\end{table}
%
\begin{figure}[h!]
\centering
\begin{tabular}{cccc}
\includegraphics[trim = 76mm 73.5mm 106.5mm 60mm, clip, width=1.8cm, height = 4.0cm]{usps_noise2_org.pdf}&
\includegraphics[trim = 76mm 73.5mm 106.5mm 60mm, clip, width=1.8cm, height = 4.0cm]{usps_noise2_corrupted.pdf} &
\includegraphics[trim = 76mm 73.5mm 106.5mm 60mm, clip, width=1.8cm, height = 4.0cm]{usps_ddae_rec.pdf} &
\includegraphics[trim = 76mm 73.5mm 106.5mm 60mm, clip, width=1.8cm, height = 4.0cm]{usps_noise2_rec.pdf} \\
(a) & (b) & (c) &(d)
\end{tabular}
\caption{The denoising results comparison on USPS alphabet (the type 2 noise). (a) original images from `A' to `Z' arranged in the top-down manner; (b) noisy images with random structures; (c) denoising results with deep denoising autoencoder; (d) denoising results with our joint model. }%It demonstrates that DDAE works well on the structural noise. It also shows that it is useful to add feedback into DDAE to enhance the results.}
\label{fig:usps}
\end{figure}

% --------------------------------------------------------------------------------------
\section{Conclusions}
In this paper, we consider the joint structural denoising and recognition problems on the handwriting images. We proposed a unified framework, which is a 3-fan deep architecture, and can learn the shared hidden representations for more complex multi-tasks. In a sense, our model can be thought as a mixture of deep autoencoder and deep feedforward neural network, which are unified in the joint framework for both reconstruction and classification tasks. The experimental results show the advantages of our model over competitive baselines on both denoising and recognition tasks.

% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------
\bibliographystyle{IEEEtran}
\bibliography{crbmbib}

\end{document}

% author names and affiliations
% use a multiple column layout for up to three different
% affiliations
%\author{\IEEEauthorblockN{Gang Chen and Sargur N. Srihari}
%\IEEEauthorblockA{Department of Computer Science and Engineering\\
%University at Buffalo, The State University of New York\\
%Buffalo, NY 14260\\
%Email: \{gangchen,srihari\}@buffalo.edu}
%%\and
%%\IEEEauthorblockN{Sargur H. Srihari}
%%\IEEEauthorblockA{Twentieth Century Fox\\
%%NY 14260, USA\\
%%Email: srihari@cedar.buffalo.edu}
%%\and
%%\IEEEauthorblockN{James Kirk\\ and Montgomery Scott}
%%\IEEEauthorblockA{Starfleet Academy\\
%%San Francisco, California 96678-2391\\
%%Telephone: (800) 555--1212\\
%%Fax: (888) 555--1212}
%}





icip2016_jointlearning.bbl
% Generated by IEEEtran.bst, version: 1.12 (2007/01/11)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
  \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl

\bibitem{Wiener64}
N.~Wiener, \emph{Extrapolation, Interpolation, and Smoothing of Stationary Time
  Series}.\hskip 1em plus 0.5em minus 0.4em\relax The MIT Press, 1964.

\bibitem{besag86}
J.~Besag, ``On the statistical analysis of dirty pictures,'' \emph{Journal of
  the Royal Statistical Society. Series B}, vol.~48, no.~3, pp. 259--302, 1986.

\bibitem{Elad06}
M.~Elad and M.~Aharon, ``Image denoising via learned dictionaries and sparse
  representation,'' in \emph{CVPR}, 2006, pp. 17--22.

\bibitem{Xie12}
J.~Xie, L.~Xu, and E.~Chen, ``Image denoising and inpainting with deep neural
  networks,'' in \emph{NIPS}, 2012.

\bibitem{LeCun89}
Y.~LeCun, B.~Boser, J.~S. Denker, D.~Henderson, R.~E. Howard, W.~Hubbard, and
  L.~D. Jackel, ``Backpropagation applied to handwritten zip code
  recognition,'' \emph{Neural Comput.}, vol.~1, no.~4, pp. 541--551, Dec. 1989.

\bibitem{Larochelle12}
\BIBentryALTinterwordspacing
H.~Larochelle, M.~Mandel, R.~Pascanu, and Y.~Bengio, ``Learning algorithms for
  the classification restricted boltzmann machine,'' \emph{J. Mach. Learn.
  Res.}, vol.~13, no.~1, pp. 643--669, Mar. 2012. [Online]. Available:
  \url{http://dl.acm.org/citation.cfm?id=2503308.2188407}
\BIBentrySTDinterwordspacing

\bibitem{Hinton06b}
G.~E. Hinton and R.~R. Salakhutdinov, ``Reducing the dimensionality of data
  with neural networks,'' \emph{Science}, vol. 313, no. 5786, pp. 504--507,
  Jul. 2006.

\bibitem{Ngiam10}
J.~Ngiam, A.~Khosla, M.~Kim, J.~Nam, H.~Lee, and A.~Y. Ng, ``Multimodal deep
  learning.'' in \emph{ICML}, 2011, pp. 689--696.

\bibitem{Hinton06a}
G.~E. Hinton, S.~Osindero, and Y.-W. Teh, ``A fast learning algorithm for deep
  belief nets,'' \emph{Neural Comput.}, vol.~18, no.~7, pp. 1527--1554, Jul.
  2006.

\bibitem{Bengio12}
Y.~Bengio, A.~Courville, and P.~Vincent, ``Representation learning: A review
  and new perspectives,'' \emph{TPAMI}, 2012.

\bibitem{Srivastava14b}
N.~Srivastava and R.~Salakhutdinov, ``Multimodal learning with deep boltzmann
  machines,'' \emph{Journal of Machine Learning Research}, vol.~15, pp.
  2949--2980, 2014.

\bibitem{RudinO94}
L.~I. Rudin and S.~Osher, ``Total variation based image restoration with free
  local constraints.'' in \emph{ICIP}.\hskip 1em plus 0.5em minus 0.4em\relax
  IEEE, 1994, pp. 31--35.

\bibitem{Tomasi98}
C.~Tomasi and R.~Manduchi, ``Bilateral filtering for gray and color images,''
  in \emph{ICCV}.\hskip 1em plus 0.5em minus 0.4em\relax Washington, DC, USA:
  IEEE Computer Society, 1998, pp. 839--.

\bibitem{Portilla03}
J.~Portilla, V.~Strela, M.~J. Wainwright, and E.~P. Simoncelli, ``Image
  denoising using scale mixtures of gaussians in the wavelet domain,''
  \emph{IEEE Trans. Image Process}, vol.~12, pp. 1338--1351, 2003.

\bibitem{Jain08}
V.~Jain and H.~S. Seung, ``Natural image denoising with convolutional
  networks.'' in \emph{NIPS}.\hskip 1em plus 0.5em minus 0.4em\relax Curran
  Associates, Inc., 2008, pp. 769--776.

\bibitem{ChenXC12}
G.~Chen, C.~Xiong, and J.~J. Corso, ``Dictionary transfer for image denoising
  via domain adaptation,'' in \emph{19th {IEEE} International Conference on
  Image Processing, {ICIP} 2012, Lake Buena Vista, Orlando, FL, USA, September
  30 - October 3, 2012}, 2012, pp. 1189--1192.

\bibitem{Gang14b}
G.~Chen and S.~N. Srihari, ``Removing structural noise in handwriting images
  using deep learning,'' in \emph{ICVGIP}, 2014.

\bibitem{Vincent10}
P.~Vincent, H.~Larochelle, I.~Lajoie, Y.~Bengio, and P.-A. Manzagol, ``Stacked
  denoising autoencoders: Learning useful representations in a deep network
  with a local denoising criterion,'' \emph{JMLR}, 2010.

\bibitem{Larochelle08}
H.~Larochelle and Y.~Bengio, ``Classification using discriminative restricted
  boltzmann machines,'' in \emph{Proceedings of the 25th International
  Conference on Machine Learning}, ser. ICML '08.\hskip 1em plus 0.5em minus
  0.4em\relax New York, NY, USA: ACM, 2008, pp. 536--543.

\bibitem{Krizhevsky12}
A.~Krizhevsky, I.~Sutskever, and G.~E. Hinton, ``Imagenet classification with
  deep convolutional neural networks,'' in \emph{Advances in Neural Information
  Processing Systems}, 2012.

\bibitem{Salakhutdinov12}
R.~Salakhutdinov and G.~E. Hinton, ``An efficient learning procedure for deep
  boltzmann machines,'' \emph{Neural Computation}, vol.~24, no.~8, pp.
  1967--2006, 2012.

\bibitem{Tang12}
Y.~Tang, R.~Salakhutdinov, and G.~Hinton, ``Robust boltzmann machines for
  recognition and denoising,'' in \emph{CVPR}.\hskip 1em plus 0.5em minus
  0.4em\relax Washington, DC, USA: IEEE Computer Society, 2012, pp. 2264--2271.

\bibitem{ChenS15}
G.~Chen and S.~N. Srihari, ``Generalized k-fan multimodal deep model with
  shared representations,'' \emph{CoRR}, vol. abs/1503.07906, 2015.

\end{thebibliography}
