Word Recognition with Deep Conditional Random Fields
\IEEEauthorblockN{Michael Shell\IEEEauthorrefmark{1
Recognition of handwritten words continues to be an important problem in document analysis and recognition. Existing approaches extract hand-engineered features from word images--which can perform poorly with new data sets. Recently, deep learning has attracted great attention because of the ability to learn features from raw data. Moreover they have yielded state-of-the-art results in classification tasks including character recognition and scene recognition. On the other hand, word recognition is a sequential problem where we need to model the correlation between characters. In this paper, we propose using deep Conditional Random Fields (deep CRFs) for word recognition. Basically, we combine CRFs with deep learning, in which deep features are learned and sequences are labeled in a unified framework. We pre-train the deep structure with stacked restricted Boltzmann machines (RBMs) for feature learning and optimize the entire network with an online learning algorithm. The proposed model was evaluated on two datasets, and seen to perform significantly better than competitive baseline models. %Word recognition is an important problem in document analysis. However, conventional approaches extract hand-engineered features from handwriting data, which in general requires years of experience and expertise. Recently, deep learning has attracted great attention because it can learn features from raw data and has yielded the state of the art results on many classification tasks, such as character recognition and scene recognition. %In this paper, we leverage the advances of deep learning, which can learn discriminative features automatically for classification problems. %Conditional Random Fields (CRFs) has been an powerful model for sequential labeling, with applications on a wide range of tasks, such as character recognition, speech recognition and natural language processing. However, how to select discriminative features is still a challenge in CRFs. Inspired by deep learning for feature representation, we incorporate it into CRFs, so that it can discover discriminative features to improve sequential classification. %On the other hand, as a sequential problem, word recognition tasks should consider the correlation between characters in order to improve accuracy. In this paper, we use deep Conditional Random Fields (deep CRFs) for word recognition. Basically, we combine CRFs with deep learning, %the inter-relationship between labels in our deep learning structure, in order to incorporate the context information from the sequential data. %our model explores CRFs %in which deep features are learned and sequences are labeled in an unified framework. %%Thus, our model is more powerful than linear Conditional Random Fields (CRFs) because the objective function learns latent non-linear features so that target labeling can be better predicted. %We pretrain the deep structure with stacked restricted Boltzmann machines (RBMs) for feature learning and optimize the whole network with online learning algorithm. We test our model on two word recognition datasets, and show that our model outperforms significantly over the completive baselines.
icip2016_deepcrf.tex

%% bare_conf.tex
%% V1.3
%% 2007/01/11
%% by Michael Shell
%% See:
%% http://www.michaelshell.org/
%% for current contact information.
%%
%% This is a skeleton file demonstrating the use of IEEEtran.cls
%% (requires IEEEtran.cls version 1.7 or later) with an IEEE conference paper.
%%
%% Support sites:
%% http://www.michaelshell.org/tex/ieeetran/
%% http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/
%% and
%% http://www.ieee.org/

%%*************************************************************************
%% Legal Notice:
%% This code is offered as-is without any warranty either expressed or
%% implied; without even the implied warranty of MERCHANTABILITY or
%% FITNESS FOR A PARTICULAR PURPOSE! 
%% User assumes all risk.
%% In no event shall IEEE or any contributor to this code be liable for
%% any damages or losses, including, but not limited to, incidental,
%% consequential, or any other damages, resulting from the use or misuse
%% of any information contained here.
%%
%% All comments are the opinions of their respective authors and are not
%% necessarily endorsed by the IEEE.
%%
%% This work is distributed under the LaTeX Project Public License (LPPL)
%% ( http://www.latex-project.org/ ) version 1.3, and may be freely used,
%% distributed and modified. A copy of the LPPL, version 1.3, is included
%% in the base LaTeX documentation of all distributions of LaTeX released
%% 2003/12/01 or later.
%% Retain all contribution notices and credits.
%% ** Modified files should be clearly indicated as such, including  **
%% ** renaming them and changing author support contact information. **
%%
%% File list of work: IEEEtran.cls, IEEEtran_HOWTO.pdf, bare_adv.tex,
%%                    bare_conf.tex, bare_jrnl.tex, bare_jrnl_compsoc.tex
%%*************************************************************************

% *** Authors should verify (and, if needed, correct) their LaTeX system  ***
% *** with the testflow diagnostic prior to trusting their LaTeX platform ***
% *** with production work. IEEE's font choices can trigger bugs that do  ***
% *** not appear when using other class files.                            ***
% The testflow support page is at:
% http://www.michaelshell.org/tex/testflow/



% Note that the a4paper option is mainly intended so that authors in
% countries using A4 can easily print to A4 and see how their papers will
% look in print - the typesetting of the document will not typically be
% affected with changes in paper size (but the bottom and side margins will).
% Use the testflow package mentioned above to verify correct handling of
% both paper sizes by the user's LaTeX system.
%
% Also note that the "draftcls" or "draftclsnofoot", not "draft", option
% should be used if it is desired that the figures are to be displayed in
% draft mode.
%
\documentclass{article}
\usepackage{spconf,amsmath,graphicx}


\usepackage{subfigure} 
\usepackage{amsmath,amsthm}
\usepackage{slashbox}
\usepackage{multirow}
\usepackage{amssymb}

\newcommand{\argmin}{\arg\!\min}
\newcommand{\argmax}{\operatornamewithlimits{argmax}}

\usepackage{hyperref}


% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}

% Title.
% ------
\title{Word Recognition with Deep Conditional Random Fields}
%
% Single address.
% ---------------
%\name{Gang Chen and Sargur N. Srihari}
%\address{Department of Computer Science and Engineering\\
%University at Buffalo, The State University of New York\\
%Buffalo, NY 14260\\
%Email: \{gangchen,srihari\}@buffalo.edu}
\name{Gang Chen$^{\dagger}$ \qquad Yawei Li$^{\star}$ \qquad Sargur N. Srihari$^{\dagger}$}

\address{$^{\dagger}$ Department of Computer Science, SUNY at Buffalo, Buffalo NY 14260\\
    $^{\star}$ School of Communication and Information Engineering \\
    University of Electronic Science and Technology of China \\Chengdu, Sichuan 611731 China}
\begin{document}
%
% paper title
% can use linebreaks \\ within to get better formatting as desired
\title{Word Recognition with Deep Conditional Random Fields }


% conference papers do not typically use \thanks and this command
% is locked out in conference mode. If really needed, such as for
% the acknowledgment of grants, issue a \IEEEoverridecommandlockouts
% after \documentclass

% for over three affiliations, or if they all won't fit within the width
% of the page, use this alternative format:
% 
%\author{\IEEEauthorblockN{Michael Shell\IEEEauthorrefmark{1},
%Homer Simpson\IEEEauthorrefmark{2},
%James Kirk\IEEEauthorrefmark{3}, 
%Montgomery Scott\IEEEauthorrefmark{3} and
%Eldon Tyrell\IEEEauthorrefmark{4}}
%\IEEEauthorblockA{\IEEEauthorrefmark{1}School of Electrical and Computer Engineering\\
%Georgia Institute of Technology,
%Atlanta, Georgia 30332--0250\\ Email: see http://www.michaelshell.org/contact.html}
%\IEEEauthorblockA{\IEEEauthorrefmark{2}Twentieth Century Fox, Springfield, USA\\
%Email: homer@thesimpsons.com}
%\IEEEauthorblockA{\IEEEauthorrefmark{3}Starfleet Academy, San Francisco, California 96678-2391\\
%Telephone: (800) 555--1212, Fax: (888) 555--1212}
%\IEEEauthorblockA{\IEEEauthorrefmark{4}Tyrell Inc., 123 Replicant Street, Los Angeles, California 90210--4321}}




% use for special paper notices
%\IEEEspecialpapernotice{(Invited Paper)}




% make the title area
\maketitle


\begin{abstract}
Recognition of handwritten words continues to be an important problem in document analysis and recognition. Existing approaches extract hand-engineered features from word images--which can perform poorly with new data sets. Recently, deep learning has attracted great attention because of the ability to learn features from raw data. Moreover they have yielded state-of-the-art results in classification tasks including character recognition and scene recognition. On the other hand, word recognition is a sequential problem where we need to model the correlation between characters. In this paper, we propose using deep Conditional Random Fields (deep CRFs) for word recognition. Basically, we combine CRFs with deep learning, in which deep features are learned and sequences are labeled in a unified framework. We pre-train the deep structure with stacked restricted Boltzmann machines (RBMs) for feature learning and optimize the entire network with an online learning algorithm. The proposed model was evaluated on two datasets, and seen to perform  significantly better than competitive baseline models.
%Word recognition is an important problem in document analysis. However, conventional approaches extract hand-engineered features from handwriting data, which in general requires years of experience and expertise. Recently, deep learning has attracted great attention because it can learn features from raw data and has yielded the state of the art results on many classification tasks, such as character recognition and scene recognition. %In this paper, we leverage the advances of deep learning, which can learn discriminative features automatically for classification problems. %Conditional Random Fields (CRFs) has been an powerful model for sequential labeling, with applications on a wide range of tasks, such as character recognition, speech recognition and natural language processing. However, how to select discriminative features is still a challenge in CRFs. Inspired by deep learning for feature representation, we incorporate it into CRFs, so that it can discover discriminative features to improve sequential classification. 
%On the other hand, as a sequential problem, word recognition tasks should consider the correlation between characters in order to improve accuracy. In this paper, we use deep Conditional Random Fields (deep CRFs) for word recognition. Basically, we combine CRFs with deep learning, %the inter-relationship between labels in our deep learning structure, in order to incorporate the context information from the sequential data. %our model explores CRFs
%in which deep features are learned and sequences are labeled in an unified framework. 
%%Thus, our model is more powerful than linear Conditional Random Fields (CRFs) because the objective function learns latent non-linear features so that target labeling can be better predicted. 
%We pretrain the deep structure with stacked restricted Boltzmann machines (RBMs) for feature learning and optimize the whole network with online learning algorithm. We test our model on two word recognition datasets, and show that our model outperforms significantly over the completive baselines. 
\end{abstract}
% IEEEtran.cls defaults to using nonbold math in the Abstract.
% This preserves the distinction between vectors and scalars. However,
% if the conference you are submitting to favors bold math in the abstract,
% then you can use LaTeX's standard command \boldmath at the very start
% of the abstract to achieve this. Many IEEE journals/conferences frown on
% math in the abstract anyway.

% no keywords


\section{Introduction}
Word recognition \cite{LeCun89,Plamondon00,Shetty07} can be formulated as a sequence labeling problem. Each word image is a set of candidate character segments obtained first by segmentation methods, then it is labeled with classifiers, such as support vector machines (SVM) and Conditional Random Fields (CRFs). Although it has been researched many years, it is still an challenge problem, considering the complex writing styles. For example, the poor quality in handwritten documents makes character boundaries hard to determine \cite{LeCun98}. What's more, even given the segmented characters in the word images, it is still hard to get satisfied results, because different people have different handwriting styles. In this paper, we attempt to address the second issue. In other words, we assume character segmentation is given, and then directly recognize the entire word without character segmentation, as \cite{Shetty07} did.%, and the recognition problem is formulated as a problem of labeling observation sequences on a large-vocabulary of words.

%In general, we need two components in the word recognition task: data representation and classifier. Conventional approaches focus more on designing features manually, which requires years of experiences in handwriting community. 
%As in many other hand- written documents, the poor quality makes character boundaries hard to determine. So we assume character segmentation is given, and then directly recognize the entire word without character segmentation, as [5] did, and the recognition problem is formulated as a problem of labelling observation sequences on a large-vocabulary of words.
Recently, deep learning has attracted great attention because it can learn features automatically from raw data, which has been thought as a vital step forward to artificial intelligence (AI). %One problem for the current deep learning models, such as convolutional neural network (CNN) \cite{Krizhevsky12} or deep belief network (DBN) \cite{Hinton06a} is that it is only for static data, not for structural output. In other words, it does not consider the correlation between characters in the word image. 
Linear CRFs, as a graphic model with correlation among labels, has been a powerful tool for sequential labeling, with applications on a wide range of tasks, such as character recognition, speech recognition and natural language processing. Moreover, as a discriminative model, it has shown advantages over generative models, i.e. HMMs on labeling sequence data \cite{Feng06,Maaten11}. Thus, we can leverage CRFs for word recognition, because it has shown promising results on a variety of handwriting datasets \cite{Feng06,Shetty07}. %However, how to select discriminative features is still a challenge in CRFs. 

Inspired by deep learning for feature learning, we unify deep learning and CRFs into one framework, so that it can discover discriminative features to improve word classification task. 
%In this paper, we leverage the advances of deep learning, which can learn discriminative features automatically for classification problems. Conditional Random Fields (CRFs) has been an powerful model for sequential labeling, with applications on a wide range of tasks, such as character recognition, speech recognition and natural language processing. 
% In this paper, we propose to tackle with word recognition with deep conditional random fields. 
%On the one hand, our deep CRFs leverages the advances of deep learning, which can learn discriminative features automatically for classification problems. On the other hand, we incorporate the context information with CRFs to improve the sequential labeling accuracy. 
Thus, our model is more powerful than linear Conditional Random Fields (CRFs) because the objective function learns latent non-linear features so that target labeling can be better predicted. Different from traditional approaches, we train our model with an independent learning stage and also use online learning to estimate model parameters. We test our method on two handwriting datasets, and show the advantages of our method over shallow CRFs models and deep learning methods significantly. 

\section{Related work}
Handwriting recognition \cite{Plamondon00} is a classical recognition problem, which has been researched for a long time. However, considering the complex cases, such as cursive styles and spurious strokes, it is far from being solved. %Off-line handwrit- ing recognition is a harder problem. Considering different writing styles and structural noise in the documents, large vocabulary recognition on modern and historical data is still a challenging task \cite{Vinciarelli03,Feng06}.

Over the past decades, many methods have beed proposed \cite{LeCun89,Hinton06b} for handwriting recognition, and good success has been achieved for small-vocabulary and highly constrained domains such as digital recognition \cite{Hinton06b}, mail sorting \cite{LeCun89,Plamondon00} and check processing \cite{Madhvanath01}. Marti et al proposed to use Hidden Markov model (HMM) to incorporate the context information for handwritten material recognition \cite{Marti01}. In their method, each character has 14 states, and words are modeled as a concatenation of these states under HMM. Later, Vinciarelli et al proposed a sliding window approach \cite{Vinciarelli03}, an high order n-gram HMM model (up to trigrams) and demonstrated promising results on large vocabulary handwritten data. Boosted decision trees \cite{Howe05} had been used on word recognition and retrieval, and yielded good retrieval performance with low word error rate. Inspired by the advantages of CRFs \cite{Lafferty01} for sequential labeling problem, several methods were put forward recently. For example, \cite{Feng06} leveraged CRFs for handwriting recognition, and demonstrated that on the whole word recognition task, linear CRFs performs better than HMM. Similarly, another CRFs with dynamical programming \cite{Shetty07} was also proposed to word recognition. 

More recently, deep learning methods, such as CNN, DBN and recurrent neural network (RNN) \cite{Hochreiter97} had gained significant attention on classification tasks. For example, Graves et al \cite{Graves08} used the bi-directional and multi-dimensional long short term memory (LSTM) \cite{Karpathy15} and yielded promising results on Arabic handwriting recognition. Furthermore, Hidden conditional random fields \cite{Maaten11} was proposed and yielded the state of the art results on the upenn OCR dataset \cite{Taskar03}. CRFs also has been combined with neural network for sequential labeling (DNN+CRFs \cite{Do10}).%Basically it assumes one hidden layer between observation and labels, in order to improve the representation power in the recognition tasks. 

%We can think the advantages of CRFs can be summarized as (1) it is a conditional model with context information; (2) it is a discriminative model, compared to the generative models, such as HMMs \cite{Feng06,Maaten11}. Thus, in this paper, we prefer to use CRFs for this task and extend it into the deep CRFs for handwriting recognition. In a sense, to compensate its weakness, we enhance its representation power with deep learning, which can learn useful features to improve the recognition accuracy \cite{Hinton06b}. We believe this is the first use of deep CRFs in this domain. We compared our deep CRFs to other baselines and showed significant improvement on word recognition accuracy. 

%Recent research on machine learning shows CRFs have advantages over HMMs on tasks which involve labelling se-quence data. On this kind of task, generative models such as HMM define a joint probability over observation and label sequences, which theoretically requires enumeration of all possible observation sequences. CRFs, as conditional undi- rected graphic models, model the conditional probabilities of label sequences given an observation sequence. In other words they don�t involve generating a testing observation sequence. Furthermore, CRFs allow arbitrary dependencies on the observation sequence and have comparable computa- tional efficiency if dependencies within state sequences are constrained. So in this paper, we investigate using CRFs for this task and compare them to HMMs and maximum entropy. We believe this is the first use of CRFs in this do- main.

%
%\begin{figure}[!t]
%\centering
%\includegraphics[width=2.5in]{myfigure}
% where an .eps filename suffix will be assumed under latex, 
% and a .pdf suffix will be assumed for pdflatex; or what has been declared
% via \DeclareGraphicsExtensions.
%\caption{Simulation Results}
%\label{fig_sim}
%\end{figure}

% Note that IEEE typically puts floats only at the top, even when this
% results in a large percentage of a column being occupied by floats.


% An example of a double column floating figure using two subfigures.
% (The subfig.sty package must be loaded for this to work.)
% The subfigure \label commands are set within each subfloat command, the
% \label for the overall figure must come after \caption.
% \hfil must be used as a separator to get equal spacing.
% The subfigure.sty package works much the same way, except \subfigure is
% used instead of \subfloat.
%
%\begin{figure*}[!t]
%\centerline{\subfloat[Case I]\includegraphics[width=2.5in]{subfigcase1}%
%\label{fig_first_case}}
%\hfil
%\subfloat[Case II]{\includegraphics[width=2.5in]{subfigcase2}%
%\label{fig_second_case}}}
%\caption{Simulation results}
%\label{fig_sim}
%\end{figure*}
%
% Note that often IEEE papers with subfigures do not employ subfigure
% captions (using the optional argument to \subfloat), but instead will
% reference/describe all of them (a), (b), etc., within the main caption.


% An example of a floating table. Note that, for IEEE style tables, the 
% \caption command should come BEFORE the table. Table text will default to
% \footnotesize as IEEE normally uses this smaller font for tables.
% The \label must come after \caption as always.
%
%\begin{table}[!t]
%% increase table row spacing, adjust to taste
%\renewcommand{\arraystretch}{1.3}
% if using array.sty, it might be a good idea to tweak the value of
% \extrarowheight as needed to properly center the text within the cells
%\caption{An Example of a Table}
%\label{table_example}
%\centering
%% Some packages, such as MDW tools, offer better commands for making tables
%% than the plain LaTeX2e tabular which is used here.
%\begin{tabular}{|c||c|}
%\hline
%One & Two\\
%\hline
%Three & Four\\
%\hline
%\end{tabular}
%\end{table}

\section{Word Recognition with Deep CRFs}
Let $D = \{ \langle {\bf x}_{i}, {\bf y}_{i}\rangle \}_{i=1}^N$ be a set of $N$ training examples. Each example is a pair of a time series $\langle {\bf x}_{i}, {\bf y}_{i}\rangle$, with ${\bf x}_{i} = \{{\bf x}_{i,1}, {\bf x}_{i,2},...,{\bf x}_{i,T_i}\}$ and ${\bf y}_{i} = \{{ y}_{i,1},{y}_{i,2},...,{y}_{i,T_i}\}$, where ${\bf x}_{i,t}  \in \mathbb{R}^d$ is the $i$-th observation at time $t$ and ${y}_{i,t}$ is the corresponding label (we indicate its encoded vector as ${\bf y}_{i,t}$ that uses a so-called 1-of-$K$ encoding).

\subsection{Linear CRFs}
Linear first-order CRFs \cite{Lafferty01} is a conditional discriminative model over the label sequence given the data 
%\begin{equation}
%\centering
%p({\bf y}_{1,...,T} | {\bf x}_{1,...,T}) = \frac{\textrm{exp}\{  E({\bf x}_{1,...,T}, {\bf y}_{1,...,T}) \}}{Z({\bf x}_{1,...,T})}
%\label{eq:eqcrf}
%\end{equation}
%where $Z({\bf x}_{1,...,T})$ is the partition function and $E({\bf x}_{1,...,T}, {\bf y}_{1,...,T})$ is the energy function given by
%\begin{align}
%E({\bf x}_{1,...,T}, {\bf y}_{1,...,T}) &= {\bf y}_1^T \boldsymbol{\pi} + {\bf y}_T^T \boldsymbol{\tau} + \sum_{t=2}^T  {\bf y}_{t-1}^T {\bf A} {\bf y}_t \nonumber \\
%&+ \sum_{t=1}^T  ({\bf x}_{t}^T {\bf W} {\bf y}_t  + {\bf c}^T {\bf y}_t )
%\label{eq:eqe}
%\end{align}
%\begin{align}
%\centering
%& p({\bf y}_{i,1},..., {\bf y}_{i,T} | {\bf x}_{i,1},..., {\bf x}_{i,T})  \nonumber \\
%= & \frac{\textrm{exp}\{ E( {\bf y}_{i,1},..., {\bf y}_{i,T}, {\bf x}_{i,1},..., {\bf x}_{i,T}) \}}{Z({\bf x}_{i,1},..., {\bf x}_{i,T})}
%\label{eq:eqcrf}
%\end{align}
\begin{equation}
\centering
p({\bf y}_{i} | {\bf x}_{i}) = \frac{\textrm{exp}\{E({\bf x}_{i}, {\bf y}_{i}) \}}{Z({\bf x}_{i})}
\label{eq:eqcrf}
\end{equation}
where $Z({\bf x}_{i})$ is the partition function and $E( {\bf x}_{i}, {\bf y}_{i})$ is the energy function given by
\begin{align}
&E( {\bf x}_{i}, {\bf y}_{i}) =  {\bf y}_{i,1}^T \boldsymbol{\pi} + {\bf y}_{i, T_i}^T \boldsymbol{\tau} \nonumber \\
+ & \sum_{t=1}^{T_i}  ({\bf x}_{i,t}^T {\bf W} {\bf y}_{i,t}  + {\bf b}^T {\bf y}_{i,t} )  + \sum_{t=2}^{T_i}  {\bf y}_{i, t-1}^T {\bf A} {\bf y}_{i,t}
\label{eq:eqe}
\end{align}
where ${\bf y}_{i,1}^T \boldsymbol{\pi}$ and ${\bf y}_{i, T_i}^T \boldsymbol{\tau}$ are the initial-state and final-state factors respectively, ${\bf b}^T {\bf y}_{i,t}$ is the bias term for labels, ${\bf A} \in \mathbb{R}^{K \times K}$ represents the state transition parameters and ${\bf W} \in \mathbb{R}^{d \times K}$ represents the classification parameter of the data-dependent term.
One of the main disadvantages of linear CRFs is the linear dependence on the raw input data term. Thus, we introduce our sequential labeling model with deep feature learning, which leverages both context information, as well as the nonlinear representations in the deep learning architecture \cite{Hinton06b}. 

\subsection{Deep CRFs}
Although it is possible to leverage the deep neural networks for structured prediction, its output space is explosively growing because of non-determined length of sequential data. Thus, we consider a compromised model, which combines CRFs and deep learning in an unified framework. %, refer Fig. (\ref{fig:fig1}). 
Thus, we propose an objective function with $L$ layers neural network structure, %\mathcal{L}({\bf x}_{i}, {\bf y}_{i}; \boldsymbol{\theta}, \boldsymbol{\omega}) 
\begin{align}
 \mathcal{L}(D; \boldsymbol{\theta}, \boldsymbol{\omega}) &= - \sum_{i=1}^N \textrm{log} p({\bf y}_{i,1},..., {\bf y}_{i,T_i}| {\bf h}_{i,1},...,  {\bf h}_{i,T_i})  \nonumber \\
 &+  \lambda_2 ||\boldsymbol{ \theta}||^2 + \lambda_3 ||\boldsymbol{\omega}||
\label{eq:eqobj}
\end{align}
where $\boldsymbol{\theta}$ and $\boldsymbol{\omega}$ are the top layer parameters and lower layer ($l = \{1,...,L-1\}$) parameters respectively, which will be explained later. The first row on the right side of the equation is from the linear CRFs  in Eq. (\ref{eq:eqcrf}), but with latent features, which depends respectively on $\boldsymbol{\theta}$ and the latent non-linear features ${\bf h}_i = \{{\bf h}_{i,1},..,{\bf h}_{i,T_i} \}$ in the coding space, with
\begin{align}
& \textrm{log} p({\bf y}_{i,1},..., {\bf y}_{i,T_i}| {\bf h}_{i,1},...,  {\bf h}_{i,T_i})   \nonumber \\
= & \sum_{t=2}^{T_i}  {\bf y}_{i, t-1}^T {\bf A} {\bf y}_{i,t} +   \sum_{t=1}^{T_i}  \big({\bf h}_{i,t}^T {\bf W} {\bf y}_{i,t}  + {\bf b}^T {\bf y}_{i,t}  \big)  \nonumber \\
+ & {\bf y}_{i,1}^T \boldsymbol{\pi} + {\bf y}_{i,T_i}^T \boldsymbol{\tau} - \textrm{log} (Z({\bf h}_{i}))
\label{eq:linearpart}
\end{align}
and non-linear mappings ${\bf h}_{i}$ is the output with $L-1$ layers neural network, s.t. 
\begin{equation}\label{eq:hidden}
{\bf h}_{i} = \underbrace{f_{L-1} \circ f_{L-2}\circ \cdot\cdot\cdot \circ f_{1}}_{L-1 \textrm{ times}}({\bf x}_{i})
\end{equation}
where $\circ$ indicates the function composition, and $f_i$ is logistic function with the weight parameter ${\bf W}_l$ respectively for $l = \{1,..,L-1\}$, refer more details in Sec. \ref{sec:learning}. With a bit abuse of notation, we denote ${\bf h}_{i,t}  = f_{1 \rightarrow (L-1)} ({\bf x}_{i,t})$. 

%The second term in the right hand side of Eq. (\ref{eq:eqobj}) is for deep feature learning, with the top layer defined as
% \begin{align}\label{eq:top}
% & \underbrace{f_{L} \circ f_{L-1}\circ \cdot\cdot\cdot \circ f_{1}}_{L\textrm{ times}} ({\bf x}_{i,t}) \nonumber \\
% =& f_{1 \rightarrow L}({\bf x}_{i,t}) =  f_{L}({\bf h}_{i,t}) =  {\bf h}_{i,t}^T {\bf W} + {\bf c}^T {\bf h}_{i,t }
%%f_{L}({\bf h}_{i,t}) = \textrm{exp}\{ {\bf h}_{i,t}^T {\bf W} + {\bf c}^T {\bf h}_{i,t }\} 
% %f_{L}({\bf h}_t) = exp\{ {\bf W}_l {\bf h}_t \}, for l \in [1,...,L-1]
%\end{align}
%where ${\bf W}$ has been defined in Eq. (\ref{eq:linearpart}), and ${\bf c}$ is the bias term. %Eq. \ref{eq:top} can be thought as the linear prediction for targets.
%Note that ${\bf W}$ is the same in both Eqs. \ref{eq:linearpart} and \ref{eq:top}. Hence, the second term can be thought as the label prediction independently without considering context information. The weighing ${\lambda_1}$ can control the balance between the first term and the second one on the RHS in Eq. (\ref{eq:eqobj}). If ${\lambda_1} \rightarrow +\infty$, then Eq. (\ref{eq:eqobj}) can be thought as the deep autoencoder \cite{Hinton06b} for classification without context information, and it can handle the cases where outputs are independent. If ${\lambda_1} \rightarrow 0$, then Eq. (\ref{eq:eqobj}) is the CRFs with non-linear deep feature learning, which generalizes the linear CRFs to learn non-linear deep mappings.
%%${\bf h}_t$ is the learned features with $L-1$ layers neural network, s.t. 
%%\begin{equation}\label{eq:eq1}
%%{\bf h}_t = \underbrace{f_{L-1} \circ f_{L-2}\circ \cdot\cdot\cdot \circ f_{1}}_{L-1 \textrm{ times}}({\bf x}_t)
%%\end{equation}

The last two terms in Eq. \ref{eq:eqobj} are for regularization on the all parameters
with $\boldsymbol{\theta} = \{ {\bf A}, {\bf W}, \boldsymbol{\pi}, \boldsymbol{\tau}, \boldsymbol{b}, \boldsymbol{c}\}$, and $\boldsymbol{\omega} = \{ {\bf W}_l |  l \in [1,..,L-1] \} $. We add the $\ell_2$ regularization to $\boldsymbol{\theta}$ as most linear CRFs does, while we have the $\ell_1$-regularized term on weight parameters $\boldsymbol{\omega}$ in the deep neural network to avoid overfitting in the learning process.

The aim of our objective function in Eq. (\ref{eq:eqobj}) is for sequential labeling, which explores both the advantages of Markov properties in CRFs and latent representations in deep learning. Firstly, our model can learn non-linear representation and label sequences with non-determined length. Secondly, our model can predict structured outputs or label sequences, while the DBN model \cite{Hinton06b} is just one label for each instance, which is independent without context information. Note that we use the first-order CRFs for clarity in Eq. \ref{eq:linearpart} and the rest of the paper, which can be easily extended for the second or high-order cases. Lastly, compared to traditional DNN+CRFs \cite{Do10}, we use an online algorithm in our deep learning model for parameter updating, which has the potential to handle large scale dataset.

\subsection{Learning}\label{sec:learning}
We take an online learning strategy in our method, which is different from traditional approaches. We use RBMs to initialize the weights for $l = \{1,..,L-1\}$ layer by layer greedily, with contrast divergence \cite{Hinton06a} (we used CD-1 in our experiments). Then we compute the sub-gradient w.r.t. $\boldsymbol{\theta}$ and $\boldsymbol{\omega}$ in the objective function, and optimize it with online learning. 

{\bf Initialization}: In our deep model, the weights from the layers $1$ to $L-1$ are ${\bf W}_l$ respectively, for $l = \{1,..,L-1\}$, and the top layer $L$ has weight ${\bf W}$. We first pre-train the $L$-layer deep structure with RBMs layer by layer greedily. 
%\begin{equation}\label{eq:eq1}
%p({\bf v},{\bf h}) \propto e^{-E({\bf v},{\bf h})}
%\end{equation}
%where the energy function is 
%\begin{equation}\label{eq:eq2}
%E({\bf v},{\bf h}) = -{\bf h}^T{\bf W}_1{\bf v} - {\bf b}^T {\bf v} - {\bf c}^T{\bf h} 
%\end{equation}
%% with parameters $\Theta_{1} = \{{\bf W}^1, {\bf b}, {\bf c}\}$. 
%And we can compute the following conditional likelihood:
%\begin{subequations}\label{eq:eq3}
%\begin{align}
%        & p({\bf v} | {\bf h}) = \prod_{i} p(v_{i} | {\bf h})\\
%        & p(v_i=1 | {\bf h}) = \mathrm{logistic}(b_{i} + \sum_{j} W_1(i,j) h_j)\\
%        & p(h_i =1 | {\bf v}) = \mathrm{logistic}(c_{i} + \sum_{j} W_1(j, i)v_j)
%\end{align}
%\end{subequations}
%where $\mathrm{logistic}(x)  = 1/(1+e^{-x})$. To learn RBM parameters, we need to optimize the negative log likelihood $-\mathrm{log}p({\bf v})$ on training data $\mathcal{D}$, the parameters updating can be calculated with a efficient stochastic descent method, namely contrastive divergence (CD) \cite{Hinton06a}.% to the negative log likelihood.
%%%$\{{\bf W}_{i}\}_{i=1}^L$
%Specifically, we think RBM is a 1-layer DBN, with weight ${\bf W}_1$. Thus, DBN can learn parametric nonlinear mapping from input ${\bf x}$ to output ${\bf h}$, $f: {\bf x} \rightarrow {\bf h}$. For example, for 1-layer DBN, we have ${\bf h} = f_1({\bf x})  = \textrm{logistic}({\bf W_{1}}^T [{\bf x}, 1])$, where we extend ${\bf x} \in \mathbb{R}^d$ into $[{\bf x},1] \in \mathbb{R}^{(d+1)}$ in order to handle bias in the non-linear mapping. %Given the output of the current layer as the input to the next layer, we can learn each layer weight greedily. 
%Note that we use the logistic function from layers $1$ to $L-1$, and the top layer is linear mapping with weight ${\bf W}$ in our deep neural network.
%
%%After learned the representation for the data, we minimize 
%%$ \lambda_1 \sum_{i=1}^N \sum_{t=1}^{T_i}  ||\underbrace{f_{L} \circ f_{L-1}\circ \cdot\cdot\cdot \circ f_{1}}_{L\textrm{ times}} ({\bf x}_{i,t}) - {\bf y}_{i,t} ||$ with L-BFGS (backpropagation is used to compute subgradient w.r.t. weights in each layer) to learn all the weights. More specifically, to learn the initial weights in the deep network, we think each instance ${\bf x}_{i,t} \in {\bf x}_{i}$ has its corresponding label ${\bf y}_{i,t} \in {\bf y}_{i}$ independently. Then, the parameters can be finetuned \cite{Hinton06b}. Note that it does not leverage the context information in this stage. %CRFs over the output in the deep network can compensate its weakness with Markov properties. 
%After initializing the weights $\boldsymbol{\omega}$ in the deep network, we will update the parameters $\boldsymbol{\theta}$ and $\boldsymbol{\omega}$ online simultaneously, which will be introduced in the following parts. 

{\bf Learning}:
In training the CRFs with deep feature learning, our aim is to minimize objective function $\mathcal{L}(D; \boldsymbol{\theta}, \boldsymbol{\omega}) $ in Eq. (\ref{eq:eqobj}). Because we introduce the deep neural network here for feature learning, the objective is not convex function anymore. However, we can find a local minimum in Eq. (\ref{eq:eqobj}). In our learning framework, we optimize the objective function with an online learn algorithm, by mixing perceptron training and stochastic gradient descent. %similar to the perceptron-based learning algorithm \cite{Gelfand10}. 

Firstly, we can calculate the (sub)gradients for all parameters. Considering different regularization methods for $\boldsymbol{\theta}$ and $\boldsymbol{\omega}$ respectively, we can calculate gradients w.r.t. them separately. As for the parameters in the negative log likelihood in Eq. \ref{eq:eqobj}, we can compute the gradients w.r.t. $\boldsymbol{\theta}$ as follows
% \begin{align}
\begin{subequations}
\begin{equation}
\frac{\mathcal{\partial L}}{\partial {\bf A}} = \sum_{i=1}^N \sum_{t=2}^{T_i} {\bf y}_{i,t-1}  ({\bf y}_{i,t})^T - \boldsymbol{\gamma}_{i,t-1}  (\boldsymbol{\gamma}_{i,t})^T;  \label{eq:grad0}
\end{equation}
%\begin{equation}
%\frac{\mathcal{\partial L}}{\partial \boldsymbol{\pi}} = \sum_{i=1}^N ({\bf y}_{i,1} - \boldsymbol{\gamma}_{i,1});  \label{eq:grad1}
%\end{equation}
%\begin{equation}
%\frac{\partial \mathcal{L}}{\partial \boldsymbol {\tau}} = \sum_{i=1}^N ({\bf y}_{i,T_i} - \boldsymbol{\gamma}_{i,T_i}); \label{eq:grad2} \\%\hat{\bf y}_{i,T_i});
%\end{equation}
%\begin{equation} 
%\frac{\mathcal{\partial L}}{\partial {\bf b}} = \sum_{i=1}^N \big( \sum_{t =1}^{T_i}( {\bf y}_{i,t} -  \boldsymbol{\gamma}_{i,t}) \big); \label{eq:grad3} \\
%%& \frac{\mathcal{\partial L}}{\partial {\bf W}} = \sum_{t=1}^T ( {\bf h}_t (y_t  -  \hat{y}_t )) +  \lambda_1 \sum_{t=1}^T (\hat{y}_t - y_t) W_{L-1} 
%\end{equation}
\begin{align}
\frac{\mathcal{\partial L}}{\partial {\bf W}} & = \sum_{i=1}^N \sum_{t =1}^{T_i}  \big( {\bf h}_{i,t} ( {\bf y}_{i,t} - \boldsymbol{\gamma}_{i,t})^T % \nonumber \\
%& +  \lambda_1  f_{1 \rightarrow (L-1)} ({\bf x}_{i,t})  ( {\bf y}_{i,t} -  \hat{\bf y}_{i,t})^T  \big)
 \end{align}
\label{eq:grad}
\end{subequations} 
where ${\bf y}_{i,t}  \in \mathbb{R}^K$ is the 1-of-$K$ encoding for labeling, $\boldsymbol{\gamma}_{i,t} \in \mathbb{R}^K$ is the vector of length $K$, which is the posterior probability for labels in the sequence and will be introduced in Sec. \ref{sec:infer}; and ${\bf h}_{i} = \{ {\bf h}_{i,1},...,{\bf h}_{i,T_i} \}$ are the latent features learned via Eq. (\ref{eq:hidden}). In the above gradients, we have ignored the gradients w.r.t. biases for convenience. Note that it is easy to derive the gradients of the $\ell_2$ regularization term w.r.t. $\boldsymbol{\theta}$ in the objective in Eq. (\ref{eq:eqobj}), which can be added to the gradients in Eq. (\ref{eq:grad}). 

As for the gradients of weights $\boldsymbol{\omega} = \{ {\bf W}_l |  l \in [1,..,L-1] \} $, we first use backpropagation to get the partial gradient in the neural network, refer to \cite{Hinton06b} for more details. Then the gradient of the $\ell_1$ term in Eq. (\ref{eq:eqobj}) can be attached to get the final gradients w.r.t. ${\bf W}_{l}$ for $l=\{1,..,L-1\}$.

Finally, we use a mixture of perceptron learning and stochastic gradient descent to optimize the objective function. %As mentioned in \cite{Maaten11}, they are various optimization methods, such as L-BFGS \cite{Byrd95,Rasmussen05}, stochastic gradient descent (SGD) and perceptron-based learning. L-BFGS as a gradient descent method, has been widely used to optimize weights in the deep structure \cite{Hinton06a}. However, it can be slow, and there are no guarantees if there are multiple local minima in the error surface. SGD and perceptron training both are the online learning algorithms by updating the parameters using the gradient induced by a single time series, so they have significant computational advantages over L-BFGS. Furthermore, perceptron-based online learning can be viewed as a special case of SGD, but it is more flexible than SGD on parameter updating (i.e. parallelization). 
In our experiments, we tried L-BFGS, but it can be easily trapped into the bad local minimum, and performs worse than other optimization methods in almost all experiments. Thus, in this work, we use perceptron-based learning for the CRF related parameters and stochastic gradient descent for the parameters in the deep structure in all our experiments. % to handle large scale and high dimensional datasets. %In addition, we find that SGD and perceptron learning converge to good solutions much faster. %perceptron-based learning algorithms outperform L-BFGS and SGD on all but the FAQ data set (on which SGD performs better).  The results of these experiments are presented in Figure 3 for perceptron and large- margin training (results obtained using L-BFGS and SGD are omitted because of space limitations).

%The perceptron training approaches aim to minimize the number of misclassified frames by updating the model parameters only when a frame is misclassified.
%If the perceptron incorrectly classifies a training example, each of the input weights is nudged a little bit in the �right direction� for that training example. In other words, we only need to update the CRF related parameters only for frames that are misclassified in each training example. To update the CRF related parameters with perceptron learning, we need to find the most violated constraints for each example. Basically, given a training example $\langle  {\bf x}_{i}, {\bf y}_{i} \rangle $, we infer its most violated labeling ${\bf y}_i^{\ast}$. If the frame is misclassified, then it directly performs a type of stochastic gradient descent on the energy gap between the observed label sequence and the predicted label sequence. Otherwise, we do not need to update the model parameters. 
Thus, for the CRF related parameters $\boldsymbol{\theta}$ in Eq. (\ref{eq:eqobj}), we first project ${\bf x}_i$ into the code ${\bf h}_{i}$ according to Eq. (\ref{eq:hidden}). Then, the updating rule takes the form below
\begin{equation}
\boldsymbol{\theta} \leftarrow  \boldsymbol{\theta}  + \eta_{\boldsymbol{\theta} }   \frac{\partial }{\partial \boldsymbol {\theta}} \big({E({\bf h}_{i}, {\bf y}_{i}) - E({\bf h}_{i}, {\bf y}_i^{\ast})} \big)
\label{eq:theta}
\end{equation}
where ${\bf y}_i^{\ast}$ is the most violated constraint in the misclassificated case, and $\eta_{\boldsymbol{\theta} }$ is a parameter step size. Note that the posterior probability $\boldsymbol{\gamma}_{i,t} \in \mathbb{R}^K$ in Eq. (\ref{eq:grad}) should be changed into the hard label assignment ${\bf y}_{i,t}^{\ast}$ in the inference stage. 

While for the weights $\boldsymbol{\omega}$ in the deep neural network, we first use backpropagation to compute the gradients, and then update it as follows
\begin{equation}
\boldsymbol{\omega} \leftarrow  \boldsymbol{\omega}  - \eta_{\boldsymbol{\omega} }   \frac{\partial \mathcal{L}}{\partial \boldsymbol {\omega}}
\label{eq:omega}
\end{equation}
where $\eta_{\boldsymbol{\omega} }$ is the step size for the parameters.
%Perceptron training is typically computation- ally more efficient because: (1) it learns good mod- els in few iterations, (2) it uses the Viterbi algorithm instead of the forward-backward algorithm, and (3) parameters only need to be updated for frames that are erroneously classified. Perceptron training ap- proaches aim to minimize the number of misclassified frames by updating the model parameters only when a frame is misclassified. This is achieved by directly performing a type of stochastic gradient descent on the energy gap between the observed label sequence and the predicted label sequence. In the case of the hidden-unit CRF, the update rule for a parameter ? ? ? = {�,?,A,W,V,b,c} takes the form
%\begin{figure*}[t!]
%\centering
%\includegraphics[trim = 37mm 0mm 0mm 0mm, clip, width=18cm]{ocr_examples.jpg}
%\caption{The samples from OCR dataset. This dataset has total 26 alphabets and total 55 different words.}
%\label{fig_ocr}
%\end{figure*}

\subsection{Inference}\label{sec:infer}
In the testing stage, the main inferential problem is to compute the most likely label sequence ${\bf y}_{1,...,T}^{\ast}$ given the data ${\bf x}_{1,...,T}$ by $\argmax_{{{\bf y}\prime}_{1,...,T}} p({{\bf y}\prime}_{1,...,T} | {\bf x}_{1,...,T} )$.%, which can be addressed similarly using the Viterbi algorithm mentioned above.
%The inference problem can be solved efficiently with Viterbi algorithm \cite{Rabiner89,Bishop06}. 

Given any new sequence ${\bf x}_{i} = \{ {\bf x}_{i,1},...,{\bf x}_{i,T_i} \}$, we first use Eq. (\ref{eq:hidden}) to compute the non-linear code ${\bf h}_{i} = \{ {\bf h}_{i,1},...,{\bf h}_{i,T_i} \}$. Then, we can do the inference as linear CRFs does by thinking ${\bf h}_{i} $ as the new obersevation. The inference problem can be
formulated as 
\begin{equation}
{\bf y}_{1,...,T}^{\ast} = \argmax_{{{\bf y}\prime}_{1,...,T_i}} p({{\bf y}\prime}_{1,...,T_i} | {\bf h}_{i} )
\end{equation}
This can be solved efficiently with Viterbi algorithm \cite{Rabiner89,Bishop06}. Note that $\boldsymbol{\gamma}_{i,t} = p( {\bf y}_{i,t}^{\ast} | {\bf h}_{i} ) $ is the posterior probability from Viterbi algorithm, and can be used in Eq. \ref{eq:grad} to calculate the gradient w.r.t. $\boldsymbol{\theta}$.


\section{Experiments} 
To test our method, we compared our method to the state of the art approaches and performed experiments on word recognition task on two widely used datasets: OCR dataset and ICDAR2003 word recognition dataset. %Below, we described the datasets we used and also the parameter setting in the experiments.

\subsection{Data sets}
1. The OCR dataset \cite{Taskar03} contains data for 6, 877 handwritten words with 55 unique words, in which each word ${\bf x}_i$ is represented as a series of handwritten characters $\{ {\bf x}_{i1},...,{\bf x}_{i, T_i} \}$. The data consists of a total of 52, 152 characters (i.e., frames), with 26 unique classes. Each character is a binary image of size $16 \times 8$ pixels, leading to a 128-dimensional binary feature vector. %The samples are shown in Fig. \ref{fig_ocr}.
%\begin{figure*}[h!]
In our experiments, we used the four data sets in \cite{Maaten11}, which are available on the author's website\footnote{\url{http://cseweb.ucsd.edu/~lvdmaaten/hucrf/Hidden-Unit_Conditional_Random_Fields.html}}.  

2.  ICDAR 2003 word recognition dataset\footnote{\url{http://algoval.essex.ac.uk/icdar/datasets/TrialTrain/word.zip}}.%, with samples shown in Fig. \ref{fig_icdar}. 
After deleting non recognizable numbers and characters, we have 1147 effective words available for training. All of these words belong to 874 classes (words), which consisted of 70 unique characters.

\subsection{Experimental Setup}
In our experiments, we randomly initialized the weight ${\bf W}$ by sampling from the normal Gaussian distribution, and all other parameters in $\boldsymbol{\theta}$ to be zero (i.e. biases $\boldsymbol{b}$ and $ \boldsymbol{c}$, and the transition matrix ${\bf A}$ all to be zero). As for $\boldsymbol{\omega} = \{ {\bf W}_l |  l \in [1,..,L-1] \} $, we initialized them with DBN, which had been mentioned before. As for the number of layers and the number of hidden units in each layer, we set differently according to the dimensionality for different datasets. In all the experiments, we used the 3-layer deep CRFs model with hidden units [400 200 100] respectively in each layer. As for context model, we used the second order potentials over characters to recognize words. %we use the 3-layer deep networks with the number of hidden units to be [400 200 100] in each layer.

\begin{table}
\centering
\resizebox{0.7\columnwidth}{!}{%
\begin{tabular}{ |l|c| }
\hline
 \multicolumn{2}{ |c| }{{\bf Word recognition error rate} (\%)} \\\cline{1-2}
 Linear-chain CRF \cite{Lafferty01} & 53.2 \\
\hline
LSTM \cite{Graves08} & 2.31 \\
\hline
DNN \cite{Hinton06b} & 18.5 \\
\hline
Hidden-unit CRF  \cite{Maaten11} &  4.62\\
\hline
Traditional DNN+CRFs \cite{Do10}& 5.19 \\
\hline
Our method  & {\bf 1.60} \\
\hline
\end{tabular}
}
\caption{The experimental comparisons on the OCR dataset. The results reveal the merits of our method, and show that our deep CRFs outperforms other methods significantly.}
\label{tab:ocr}
\end{table}

Evaluation: we use word error rate to measure all methods in 10-fold cross-validation. %Because L-BFGS can be easily trapped into local minimum, we used perceptron training \cite{Gelfand10} to estimate the CRF related parameters and stochastic gradient descent to learn the weights in the deep neural network. 
In all experiments with perceptron learning, we did not use regularization terms. In other words, we set $\lambda_2 =0$. And $\lambda_3= 2\times10^{-4}$ for weights in the deep network. For each dataset, we followed the protocol in \cite{Maaten11} and divided it into $10$ folds (9 folds as the training set, and the rest as the testing/validation set), and performed 100 full sweeps through the training data, to update the model parameters. We tuned the base step size based on the error on a small held-out validation set. From the base step size, we computed parameter-specific step sizes $\eta_{\boldsymbol{\theta} }$ and $\eta_{\boldsymbol{\omega} }$ as suggested by \cite{Gelfand10}. %


In Table \ref{tab:ocr}, we compared the performance of our method with the performance of competing models on the OCR dataset. Our method yields a generalization word error of $1.6\%$, while the best performance of other methods is $2.31\%$. It also demonstrates that our model is significantly better than other methods, and the deep structure is definitely helpful than the shadow models, such as hidden CRFs. In addition, our method is significantly better than traditional DNN+CRFs. Thus, as for our model, it shows that the learning stage is helpful and can improve the accuracy significantly. %method The table reveals that hidden-unit CRFs outperform a range of state-of-the- art models and systems on all three tasks.


We also tested our method on the ICDAR2003 dataset. In our experiment, we first split the words into characters, and then we tested our deep CRFs over the characters. As for word segmentation, we use an naive approach (more sophisticated word segmentation methods will be definitely helpful in this case). Basically, given the word image, we know the number of characters in this image according to its label. Thus, we split the word image equally by dividing the number of characters. Since the sizes of all images are different, we need to resize all characters into the same dimension. Hence, we computed the width-height ratio for each character (which is the split one from its corresponding word image) on all the dataset, and then get the average size for all characters. The result character size is $65\times40$, and we reshape it to a binary vector with 2600 dimension (Note that we binarize all the images or characters). Table \ref{tab:icdar2003} shows the word recognition error rate on ICDAR2003 dataset with five folder cross validation. Again, it demonstrates that our method outperforms CRFs related models and deep learning methods. It also indicates that our model is better than traditional DNN+CRFs. %Note that we only used a simple character segmentation here, and sophisticated word segmentation will
\begin{figure}[!t]
\centering
\includegraphics[trim = 50mm 95mm 40mm 95mm, clip, width=8.5cm]{icdar_example.pdf}
\caption{The samples from the ICDAR 2003 word recognition dataset. The subset in our experiment has total 70 unique characters and 874 words.}
\label{fig_icdar}
\end{figure}

\begin{table}
\centering
\resizebox{0.7\columnwidth}{!}{%
\begin{tabular}{ |l|c| }
\hline
 \multicolumn{2}{ |c| }{{\bf word recognition (error rate)} (\%)} \\\cline{1-2}
 Linear-chain CRF \cite{Lafferty01} & 97.6 \\
\hline
DNN \cite{Hinton06b} & 41.8 \\
\hline
Hidden-unit CRF  \cite{Maaten11} &  62.7\\
\hline
 Traditional DNN+CRFs \cite{Do10} & 82.6 \\
\hline
Our method & {\bf 40.2} \\
\hline
\end{tabular}
}
\caption{The experimental comparisons on the ICDAR2003 word recognition dataset. The results reveal that our deep CRFs is effective for word recognition.}
\label{tab:icdar2003}
\end{table}

\section{Conclusion}
In this paper, we propose a deep conditional random fields (deep CRFs) for word classification problems. %, which leverages the advances of both deep learning and conditional random fields. Although CRFs has been an powerful tool for sequential labeling. However, how to select discriminative and non-linear features is still a challenge in CRFs. Inspired by deep learning for feature representation, we unify deep learning and CRFs together for both feature learning and sequential labeling. 
Compared to traditional DNN+CRFs, we propose a mixture online learning algorithm: perceptron training for CRFs parameters and stochastic gradient descent for low level weights in the deep structure. We update parameters in an online fashion, which makes it possible to apply our model to large scale datasets. We tested our methods on widely used word recognition datasets, and show that our deep CRFs is effective compared to other shallow CRFs and deep learning methods. %In the further work, we will consider more sophisticated word segmentation and 
% conference papers do not normally have an appendix

% use section* for acknowledgement
%\section*{Acknowledgment}
%The authors would like to thank...


% trigger a \newpage just before the given reference
% number - used to balance the columns on the last page
% adjust value as needed - may need to be readjusted if
% the document is modified later
%\IEEEtriggeratref{8}
% The "triggered" command can be changed if desired:
%\IEEEtriggercmd{\enlargethispage{-5in}}

% references section

% can use a bibliography generated by BibTeX as a .bbl file
% BibTeX documentation can be easily obtained at:
% http://www.ctan.org/tex-archive/biblio/bibtex/contrib/doc/
% The IEEEtran BibTeX style support page is at:
% http://www.michaelshell.org/tex/ieeetran/bibtex/
%\bibliographystyle{IEEEtran}
% argument is your BibTeX string definitions and bibliography database(s)
%\bibliography{IEEEabrv,../bib/paper}
%
% <OR> manually copy in the resultant .bbl file
% set second argument of \begin to the number of references
% (used to reserve space for the reference number labels box)
%\begin{thebibliography}{1}
%\bibitem{IEEEhowto:kopka}
%H.~Kopka and P.~W. Daly, \emph{A Guide to \LaTeX}, 3rd~ed.\hskip 1em plus
%  0.5em minus 0.4em\relax Harlow, England: Addison-Wesley, 1999.
%\end{thebibliography}


\bibliography{hrbib}
\bibliographystyle{IEEEtran}


% that's all folks
\end{document}



icip2016_deepcrf.bbl
% Generated by IEEEtran.bst, version: 1.12 (2007/01/11)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
  \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl

\bibitem{LeCun89}
Y.~LeCun, B.~Boser, J.~S. Denker, D.~Henderson, R.~E. Howard, W.~Hubbard, and
  L.~D. Jackel, ``Backpropagation applied to handwritten zip code
  recognition,'' \emph{Neural Comput.}, vol.~1, no.~4, pp. 541--551, Dec. 1989.

\bibitem{Plamondon00}
R.~Plamondon and S.~N. Srihari, ``On-line and off-line handwriting recognition:
  A comprehensive survey,'' \emph{IEEE Trans. Pattern Anal. Mach. Intell.},
  vol.~22, no.~1, pp. 63--84, Jan. 2000.

\bibitem{Shetty07}
S.~Shetty, H.~Srinivasan, and S.~N. Srihari, ``Handwritten word recognition
  using conditional random fields.'' in \emph{ICDAR}.\hskip 1em plus 0.5em
  minus 0.4em\relax IEEE Computer Society, 2007, pp. 1098--1102.

\bibitem{LeCun98}
Y.~Lecun, L.~Bottou, Y.~Bengio, and P.~Haffner, ``Gradient-based learning
  applied to document recognition,'' in \emph{Proceedings of the IEEE}, 1998,
  pp. 2278--2324.

\bibitem{Krizhevsky12}
A.~Krizhevsky, I.~Sutskever, and G.~E. Hinton, ``Imagenet classification with
  deep convolutional neural networks,'' in \emph{Advances in Neural Information
  Processing Systems}, 2012.

\bibitem{Hinton06a}
G.~E. Hinton, S.~Osindero, and Y.-W. Teh, ``A fast learning algorithm for deep
  belief nets,'' \emph{Neural Comput.}, vol.~18, no.~7, pp. 1527--1554, jul
  2006.

\bibitem{Feng06}
S.~Feng, R.~Manmatha, and A.~McCallum, ``Exploring the use of conditional
  random field models and hmms for historical handwritten document
  recognition,'' in \emph{Proceedings of the Second International Conference on
  Document Image Analysis for Libraries}, ser. DIAL '06.\hskip 1em plus 0.5em
  minus 0.4em\relax Washington, DC, USA: IEEE Computer Society, 2006, pp.
  30--37.

\bibitem{Maaten11}
L.~van~der Maaten, M.~Welling, and L.~K. Saul, ``Hidden-unit conditional random
  fields,'' in \emph{AISTATS}, 2011, pp. 479--488.

\bibitem{Hinton06b}
G.~E. Hinton and R.~R. Salakhutdinov, ``Reducing the dimensionality of data
  with neural networks,'' \emph{Science}, vol. 313, no. 5786, pp. 504--507,
  Jul. 2006.

\bibitem{Madhvanath01}
S.~Madhvanath and V.~Govindaraju, ``The role of holistic paradigms in
  handwritten word recognition,'' \emph{IEEE Trans. Pattern Anal. Mach.
  Intell.}, vol.~23, no.~2, pp. 149--164, Feb. 2001.

\bibitem{Marti01}
U.-V. Marti and H.~Bunke, ``Using a statistical language model to improve the
  performance of an hmm-based cursive handwriting recognition systems,''
  \emph{Jnl. of Pattern Recognition and Artifi cal Intelligence}, pp. 65--90,
  2001.

\bibitem{Vinciarelli03}
A.~Vinciarelli, S.~Bengio, and H.~Bunke, ``Offline recognition of large
  vocabulary cursive handwritten text,'' in \emph{Proceedings of International
  Conference on Document Analysis and Recognition (ICDAR)}, 2003.

\bibitem{Howe05}
N.~R. Howe, T.~M. Rath, and R.~Manmatha, ``Boosted decision trees for word
  recognition in handwritten document retrieval,'' in \emph{Proceedings of the
  28th Annual International ACM SIGIR Conference on Research and Development in
  Information Retrieval}, ser. SIGIR '05.\hskip 1em plus 0.5em minus
  0.4em\relax New York, NY, USA: ACM, 2005, pp. 377--383.

\bibitem{Lafferty01}
J.~D. Lafferty, A.~McCallum, and F.~C.~N. Pereira, ``Conditional random fields:
  Probabilistic models for segmenting and labeling sequence data,'' in
  \emph{ICML}, 2001, pp. 282--289.

\bibitem{Hochreiter97}
S.~Hochreiter and J.~Schmidhuber, ``Long short-term memory,'' \emph{Neural
  Comput.}, vol.~9, no.~8, pp. 1735--1780, Nov. 1997.

\bibitem{Graves08}
A.~Graves and J.~Schmidhuber, ``Offline handwriting recognition with
  multidimensional recurrent neural networks.'' in \emph{NIPS}, D.~Koller,
  D.~Schuurmans, Y.~Bengio, and L.~Bottou, Eds.\hskip 1em plus 0.5em minus
  0.4em\relax Curran Associates, Inc., 2008, pp. 545--552.

\bibitem{Taskar03}
B.~Taskar, C.~Guestrin, and D.~Koller, ``Max-margin markov networks,'' in
  \emph{NIPS}.\hskip 1em plus 0.5em minus 0.4em\relax MIT Press, 2003.

\bibitem{Rabiner89}
L.~R. Rabiner, ``A tutorial on hidden markov models and selected applications
  in speech recognition,'' in \emph{PROCEEDINGS OF THE IEEE}, 1989, pp.
  257--286.

\bibitem{Bishop06}
C.~M. Bishop, \emph{Pattern Recognition and Machine Learning}.\hskip 1em plus
  0.5em minus 0.4em\relax Secaucus, NJ, USA: Springer-Verlag New York, Inc.,
  2006.

\bibitem{Do10}
T.~M.~T. Do and T.~Arti�res, ``Neural conditional random fields.'' in
  \emph{AISTATS}, ser. JMLR Proceedings, Y.~W. Teh and D.~M. Titterington,
  Eds., vol.~9.\hskip 1em plus 0.5em minus 0.4em\relax JMLR.org, 2010, pp.
  177--184.

\bibitem{Karpathy15}
A.~Karpathy and F.-F. Li, ``Deep visual-semantic alignments for generating
  image descriptions.'' in \emph{CVPR}.\hskip 1em plus 0.5em minus 0.4em\relax
  IEEE, 2015, pp. 3128--3137.

\bibitem{Gelfand10}
A.~Gelfand, Y.~Chen, L.~van~der Maaten, and M.~Welling, ``On herding and the
  perceptron cycling theorem,'' in \emph{Advances in Neural Information
  Processing Systems 23}, J.~Lafferty, C.~K.~I. Williams, J.~Shawe-Taylor,
  R.~Zemel, and A.~Culotta, Eds., 2010, pp. 694--702.

\end{thebibliography}
