Real-World Font Recognition Using Deep Network and Domain Adaptation
Zhangyang Wang \& Thomas S. Huang
Beckman Institute for Advanced Science and Technology
University of Illinois at Urbana-Champaign
Urbana
IL 61801
USA
\{zwang119
t-huang1\
We address a challenging fine-grain classification problem: recognizing a font style from an image of text. In this task, it is very easy to generate lots of rendered font examples but very hard to obtain real-world labeled images. This real-to-synthetic domain gap caused poor generalization to new real data in previous methods (\cite{LFE). In this paper, we refer to Convolutional Neural Networks, and use an adaptation technique based on a Stacked Convolutional Auto-Encoder that exploits unlabeled real-world images combined with synthetic data. The proposed method achieves an accuracy of higher than 80\% (top-5) on a real-world dataset.
iclr2015.tex
\documentclass{article} % For LaTeX2e
\usepackage{iclr2015,times}
\usepackage{hyperref}
\usepackage{url}
\usepackage{times}
\usepackage{epsfig}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{amsmath}
\usepackage{tabularx}
\usepackage{amssymb}
\usepackage{mathrsfs}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{bm}


%\title{Autoencoder-Based Domain Adaptation for Real-World Font Recognition}
\title{Real-World Font Recognition Using Deep Network and Domain Adaptation}


\author{
Zhangyang Wang \& Thomas S. Huang\\
Beckman Institute for Advanced Science and Technology \\
University of Illinois at Urbana-Champaign\\
Urbana, IL 61801, USA \\
\texttt{\{zwang119, t-huang1\}@illinois.edu} \\
\And
Jianchao Yang \& Hailin Jin \& Eli Shechtman \&  Aseem Agarwala \& Jonathan Brandt\\
Adobe Research \\
San Jose, CA, 95110\\
\texttt{\{jiayang, hljin, elishe, asagarwa, jbrandt\}@adobe.com} \\
}


% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.

\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}

\iclrfinalcopy % Uncomment for camera-ready version
%\iclrconference % Uncomment if submitted as conference paper instead of workshop

\begin{document}


\maketitle

\begin{abstract}
We address a challenging fine-grain classification problem: recognizing a font style from an image of text. In this task, it is very easy to generate lots of rendered font examples but very hard to obtain real-world labeled images. This real-to-synthetic domain gap caused poor generalization to new real data in previous methods (\cite{LFE}). In this paper, we refer to Convolutional Neural Networks, and use an adaptation technique based on a Stacked Convolutional Auto-Encoder that exploits unlabeled real-world images combined with synthetic data. The proposed method achieves an accuracy of higher than 80\% (top-5) on a real-world dataset.
\end{abstract}


\input{sec12}
\input{sec3}
%\input{sec4}
\input{sec5}
%\input{sec6}

\bibliography{iclr2015}
\bibliographystyle{iclr2015}


\end{document}

sec12.tex
\section{Introduction}

This paper studies font recognition, i.e. identifying a particular typeface given an image of a text fragment. To apply machine learning to this problem, we require realistic text images with ground truth font labels.  However, such data is scarce and expensive to obtain, since it requires a high level of domain expertise which is out of reach of most people. Therefore, it is infeasible to collect a sufficient set of real-world training images. One way to overcome the training data challenge is to synthesize the training set by rendering text fragments for all the necessary fonts. However, we must face the domain mismatch between synthetic and real-world text images (\cite{LFE}). Characters in real-world images are spaced, stretched and distorted in numerous ways.  In (\cite{LFE}), the authors tried to overcome this difficulty by adding different degradations to synthetic data.  In the end, introducing all possible real-world degradations into the training data is infeasible. 

%, and a higher accuracy is still expected for practice use.

% Different from many previous transfer learning works \cite{Ng, DeCaf}, the domain mismatch problem we are facing occurs on the low-to-middle level features in the first place, i.e., there is a big mismatch between synthetic and real-world data on the lower-level statistics. For instance, synthetic edges can be modeled by abstract step functions while real-world edges are typically corrupted by blurring, noise and aliasing. Actually, that represents a big class of domain mismatch problems we are facing in many other applications as well, such as real-world face recognition from rendered images or sketches \cite{face}, recognizing characters in real-scene with synthetic training \cite{OCR}, human pose estimation with synthetic images generated from 3D human body models \cite{pose}.

We address this domain mismatch problem in font recognition, by further leveraging a large corpus of synthetic data to train a Convolutional Neural Network (CNN), while introducing an adaptation technique based on Stacked Convolutional Auto-Encoder (SCAE) with the help of unlabeled real-world images. The proposed method reaches an impressive performance on real-world test images.

 

%using a stacked convolutional autoencoder (SCAE) to learn low-to-middle level features that minimize the gap between synthetic and real world data.  A convolutional neural network (CNN) classifier is trained on the learned features to recognize font styles from images.

%Figure~\ref{fig:show} (a) overall pipeline of our DeepFont system [\jianchao{figure to be changed}].

%Fig.~\ref{fig:show} (b) shows a few real-world samples, coming with high levels of background clutters, size and ratio variations, and also perspective distortions, which can be correctly recognized by our DeepFont system.
%
%
%
%
%\begin{figure*}[htbp]
%\centering
%\begin{minipage}{0.90\textwidth}
%\centering \subfigure[] {
%\includegraphics[width=\textwidth]{1a_training.png}
%}\end{minipage}
%\\
%\begin{minipage}{0.90\textwidth}
%\centering \subfigure [] {
%\includegraphics[width=\textwidth]{1b.png}
%}\end{minipage}
%\caption{(a) The training pipeline of DeepFont system; (b) the correctly recognized real-world text images using DeepFont.}
%\label{fig:show}
%\end{figure*}



%focusing on small number of font classes on scanned document images that typically have high quality, in terms of resolution and contrast, and low geometric distortion. Consequently, tasks such as binarization, character segmentation, connected component analysis, geometric alignment, and OCR can be robustly applied. Building on these image processing steps, global texture analysis  \cite{15, 24} high-order statistical features \cite{2}, and typographical features \cite{8, 17} have been exploited for font recognition. However, such scanned image-based techniques are less applicable to our VFR problem, which needs to be effective even on very short strings or single words from noisy web images or photos taken with mobile devices. In particular, photos in the wild suffer from noise, blur, perspective distortions, as well as complex interactions of content with background. Moreover, limited to the scope of scanned documents, they have only investigated the problem on a scale of tens of fonts.



% Why difficult
%To address VFR, one shall realize the following specific challenges, in add-on to those typical ones in general recognition, such as handling variations in real-world images.  First, the space of possible fonts is quite large. Most computers are now equipped with hundreds of fonts, and online repositories provide hundreds of thousands, which easily exceeds the scale of most image categorization datasets in computer vision. Second, VFR itself is inherently dynamic and open-ended in that new font classes need to be continually added to the system. It is therefore critical that the algorithm is able to adapt to new classes with low added cost, and to scale elegantly with the number of classes. Third, there is usually no obvious difference between two categories; two may only differ slightly (letter endings, weights, slopes, and so on, and such differences are character-dependent. Last but not least, while the popular machine learning techniques are data-driven, collecting real-world data for a large collection of font classes turns out to be extremely difficult because many attainable real-world text images do not have font label information, and the error-prone font labeling task requires font
%expertise that is out of reach of most people.

% Contribution (DL...)

%(b) display a screenshot of a GUI software we implemented by the DeepFont model, where given the input text image in the top window, the software could list most similar fonts available in current operation system in the bottom window, in a high-to-low order of similarity. Note the typed-in text in the middle window is only for rendering and visual comparison purpose, and is not utilized to help recognition. 
sec3.tex
\section{Model}

%The first and second convolutional layers (Conv. Layer 1 and Conv. layer 2) are totally identical to the first two convolutional layers of SCAE as in Fig. \ref{fig:SAE}.

%Convolutional neural networks (CNNs) have recently shown an explosive popularity partially due to its success in image classification \cite{imagenet}.

Our basic CNN architecture is similar to the popular ImageNet CNN structure in (\cite{imagenet}), as depicted in Fig. \ref{fig:CNN}. The numbers along with the network pipeline specify the dimensions of outputs of corresponding layers.  When the CNN model trained fully on a synthetic dataset, it witnesses a significant performance drop when testing on real-world data, compared to when applied to another synthetic validation set. This also happens with other models such as in \cite{LFE}, which uses training and testing sets of similar properties to ours. This alludes to discrepancies between the distributions of synthetic and and real-world examples.

%On the VFR\_syn\_train dataset, it takes approximately 20 hours to train the above-described CNN using one single GPU.


 \begin{figure*}[htbp]
\centering
\begin{minipage}{0.90\textwidth}
\centering {
\includegraphics[width=\textwidth]{2.png}
}\end{minipage}
\caption{The CNN architecture, and its decomposition marked by different colors ($N$=8, $K$=2).}
\label{fig:CNN}
\end{figure*}


%We trained our models on using the code-convnet code developed by Alex. et. al. \cite{imagenet}. The Conv. Layer 1 and 2 are imported from the first and second convolutional layers of the SCAE. During CNN training, we fix the first two layers, which is equivalent to using them as an off-line designed feature extractor. The subsequent layers (starting from Conv. Layer 3) will be trained based on the output features, which not only depend on labeled training data but also benefit from unlabeled data, both synthetic and real-world.


%The performance is also possible to benefit from more sophisticated models on more massive data; however, they also bring much higher model complexity and the potential risk of overfitting.

Tradition approaches to handle this gap include pre-processing steps applied on the training and/or testing data (\cite{LFE}). The domain adaptation method in \cite{sentiment} extracts low-level features that represent both the synthetic and real-world data, based on a stacked auto-encoder (SAE). We extend the method in \cite{sentiment} to decompose the $N$ basic CNN layers into two sub-network parts. The first $K$ layers accounts for extracting low-level visual features shared by both synthetic and real-world data, and will be learned in a unsupervised way, using unlabeled data from both domains. The remaining $N-K$ layers accounts for learning higher-level discriminative features for classification. It will be trained in a supervised way on top of the first part, using labeled data from the synthetic domain only.

To train the first $K$ layers, we exploit a Stacked Convolutional Auto-Encoder (SCAE) (\cite{SCAE}). Its first two convolutional layers have an identical topology to the first two layers in Fig. \ref{fig:CNN}. Moreover, we set its first and second half to be mirror-symmetrical. The cost function is the mean squared error (MSE) between the input and reconstructed patches. After SCAE is learned, its Conv. Layers 1 and 2 are imported to the CNN in Fig. \ref{fig:CNN}. We adopt the SCAE implementation by \cite{paine2014analysis}.

%We visualize the convolutional kernels learned in the Conv. Layer 1 of SCAE, and compare it with the counterparts in CNN, as in Fig. \ref{fig:filters}. \textbf{Space Holder: a paragraph and a figure coming soon}

We also find that applying label-preserving data augmentations to synthetic training data helps reduce the domain mismatch. \cite{LFE} added moderate distortions and corruptions, including noise, blur, rotations and shading effects. In addition, we also vary the character spacings and aspect ratios when rendering training data. Note that these steps are not useful for the method in \cite{LFE} because it exploits very localized features, but they are very helpful in our case.

%We take the real-world example from Fig. \ref{fig:per} (a), and synthesize a series of images, all with the same text, but becoming gradually more visually similar to their real-world counterpart. Specially, (a) is synthesized with standards spacing and ratio setting and no perturbations; (b) is (a) with perturbations 1-4 added; (c) synthesized with spacing and aspect ratio customized to be identical to those of Fig. \ref{fig:per} (a); (d) adds perturbations 1-4 to (c). We input images (a)-(d) through DeepFont CAE\_PR (the best model of DeepFont; to be introduced in Section 5), and obtain the layer-wise intermediate outputs for each image. For each image (a) to (d), we normalize their layer-wise outputs, by calculating the relative MSEs on each layer's output, to the output of the same layer by feeding Fig. \ref{fig:per} (a) through the same DeepFont model.  Fig. \ref{fig:pre} (e) shows that the introduced perturbations, especially the introduced spacing and aspect ratio changes, reduce significantly the gap between real-world and synthetic features.
%%we prove their effects on recognizable features, by examining the similarity of the intermediate features between synthetic images with real-world counterparts.
%\begin{figure}[htbp]
%\centering
%\begin{minipage}{0.25\textwidth}
%\centering \subfigure[Synthetic, none] {
%\includegraphics[width=\textwidth]{R0.png}
%}
%\centering \subfigure[Synthetic, 1-4] {
%\includegraphics[width=\textwidth]{R2_blur.png}
%}\end{minipage}
%\begin{minipage}{0.25\textwidth}
%\centering \subfigure[Synthetic, 5-6] {
%\includegraphics[width=\textwidth]{R1.png}
%}
%\centering \subfigure[Synthetic, 1-6] {
%\includegraphics[width=\textwidth]{R3_blur.png}
%}\end{minipage}
%\begin{minipage}{0.45\textwidth}
%\centering \subfigure[Relative CNN layer-wise responses] {
%\includegraphics[width=\textwidth]{layer.png}
%}\end{minipage}
%\caption{The effects of data preparation steps. (a) synthetic image with the same text and no perturbation; (b) synthetic image with perturbations 1-4; (c) synthetic image with perturbations 5-6; (d) synthetic image with perturbations 1-6; (f) the comparison of layer-wise responses of (a)-(d) through DeepFont CAE\_PR, in relative to those of (a).}
%\label{fig:pre}
%\end{figure}







sec5.tex
\section{Experiments}

We implemented and evaluated the local feature embedding-based algorithm (LFE) in (\cite{LFE}) as a baseline, and compare it with our model. A SCAE is first trained on a large collection of both synthetic data and unlabeled real world data, and then exports the first $K = 2$ convolutional layers. The next $N - K$ layers are trained on labeled synthetic data covering 2,383 classes. That makes our problem quite fine-grain. Testing is conducted on the the VFRWild325 dataset used by (\cite{LFE}), in term of top-1 and top-5 classification errors. Our model achieves 38.15\% in top-1 error and 20.62\% in top-5, which outperforms 6\% and 10\% over LFE, respectively. 


%Fig. \ref{fig:win} shows two successful VFR examples using DeepFont CAE\_AR. Given the real-world images, top-5 recognition results are listed.

%% \begin{figure}[htbp]
%% \centering
%% \begin{minipage}{0.45\textwidth}
%% \centering \subfigure[] {
%% \includegraphics[width=\textwidth]{5b.png}
%% }\end{minipage}
%% \begin{minipage}{0.50\textwidth}
%% \centering \subfigure[] {
%% \includegraphics[width=\textwidth]{5c.png}
%% }\end{minipage}
%% \caption{Successful VFR examples using DeepFont CAE\_AR. The top row are query images from VFR\_real\_test  dataset. Below each query, the results (left column: font classes; right column: images rendered with the corresponding font classes) are listed in a high-to-low order in term of likelihoods predicted by the model. The groundtruth results are marked out by the red boxes.}
%% \label{fig:win}
%% \end{figure}


%Methods &  VFRWild325 Top-1 error &  VFRWild325 Top-5 error  & VFR\_real\_test Top-1 error & VFR\_real\_test Top-5 error \\
%\hline
%LFE  \cite{LFE}  & 44.13 & 30.25 & NA & NA \\
%\hline
%DeepFont S  & 64.60 & 57.23 & 57.51 & 50.76 \\
%\hline
%DeepFont P  & 43.10 & 22.47 & 33.30 & 20.72 \\
%\hline
%DeepFont CAE & 38.15 & 20.62 & 28.58 & 18.21 \\
%\hline


%\subsection{Font Similarity based on DeepFont}
%
%
%%The authors in \cite{Exploratory} propose to select fonts based on online crowdsourced attributes, While the list of font attributes cannot be really exhaustive, their experimental results suggest that font evaluation is highly subjective for both novices and professionals.
%
%
%There are a variety of font selection tasks with different goals and requirements. One designer may wish to match a font to the style of a particular image. Another may wish to find a free font which looks similar to a commercial font such as Helvetica. A third may simply be exploring a large set of fonts such as Adobe TypeKit or Google Web Fonts. Exhaustively exploring the entire space of fonts using an alphabetical listing is unrealistic for most users. The authors in \cite{Exploratory} propose to select fonts based on online crowdsourced attributes, and explore \textit{font similarity}, from which a user is enabled to explore other visually similar fonts given a specific font. The font similarity metric is very helpful for font selection, organization, browsing, and suggestion.
%
%Based on our DeepFont system, we are able to build up a metric of font similarity.  We use the $4096 \times 1$ output of the second fully connected layer in Fig. \ref{fig:CNN}, as the high-level feature vectors describing font visual appearances. We then extract such features from all samples in VFR\_syn\_val Dataset, obtaining 100 feature vectors per class. Next for each class, the 100 feature vectors is averaged to a representative vector, leading to 2,383 such representative vectors for all font classes. Finally, we calculate the Euclidean distance between the representative vectors of two font classes as their similarity metric. Visualized examples are demonstrated in Fig. \ref{fig:graph}. Note that although the result fonts often belong to different font families from the query, they are visually and stylistically similar.
%
%%For each example, we first show the query image of a known font class; we then show the the most similar fonts rendered with the same text.  Note that although the result fonts often belong to different font families from the query, they are visually and stylistically similar.
%
%%Although our method does not numerically perform as well as \cite{Exploratory}, our method performs qualitatively better when we look at the top-10 most similar fonts for a wide range of query fonts. We have shown our results to the authors of \cite{Exploratory}, and they agree with this conclusion.
%
%
%
%
%\begin{figure}[htbp]
%\centering
%\begin{minipage}{0.40\textwidth}
%\centering \subfigure[] {
%\includegraphics[width=\textwidth]{example1.pdf}
%}\end{minipage}
%\begin{minipage}{0.40\textwidth}
%\centering \subfigure[] {
%\includegraphics[width=\textwidth]{example2.pdf}
%}\end{minipage}
%\caption{Two examples of the font similarity. For each one, the top is the query image, and the renderings with the top most similar fonts are returned as results.}
%\label{fig:graph}
%\end{figure}

iclr2015.bbl
\begin{thebibliography}{5}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
  \providecommand{\doi}[1]{doi: #1}\else
  \providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi

\bibitem[Chen et~al.(2014)Chen, Yang, Jin, Brandt, Shechtman, Agarwala, and
  Han]{LFE}
Chen, G., Yang, J., Jin, H., Brandt, J., Shechtman, E., Agarwala, A., and Han,
  T.~X.
\newblock Large-scale visual font recognition.
\newblock In \emph{Proceedings of CVPR}, pp.\  3598--3605. IEEE, 2014.

\bibitem[Glorot et~al.(2011)Glorot, Bordes, and Bengio]{sentiment}
Glorot, X., Bordes, A., and Bengio, Y.
\newblock Domain adaptation for large-scale sentiment classification: A deep
  learning approach.
\newblock In \emph{Proceedings of ICML}, pp.\  513--520, 2011.

\bibitem[Krizhevsky et~al.(2012)Krizhevsky, Sutskever, and Hinton]{imagenet}
Krizhevsky, A., Sutskever, I., and Hinton, G.~E.
\newblock Imagenet classification with deep convolutional neural networks.
\newblock In \emph{Proceedings of NIPS}, pp.\  1097--1105, 2012.

\bibitem[Masci et~al.(2011)Masci, Meier, Cire{\c{s}}an, and Schmidhuber]{SCAE}
Masci, J., Meier, U., Cire{\c{s}}an, D., and Schmidhuber, J.
\newblock Stacked convolutional auto-encoders for hierarchical feature
  extraction.
\newblock In \emph{Proceedings of ICANN}, pp.\  52--59. Springer, 2011.

\bibitem[Paine et~al.(2014)Paine, Khorrami, Han, and Huang]{paine2014analysis}
Paine, T., Khorrami, P., Han, W., and Huang, T.~S.
\newblock An analysis of unsupervised pre-training in light of recent advances.
\newblock \emph{arXiv preprint arXiv:1412.6597}, 2014.

\end{thebibliography}
