Improving Document Clustering by Eliminating Unnatural Language
Author 1
...
Author n
% Address line
...
Address line
Author 1
Address line
...
Address line % \And ... \And % Author n
Address line
...
Address line
Author 1
Address line
...
Address line % \AND % Author 2
Address line
...
Address line \And % Author 3
Address line
...
Address line
Myungha Jang$^{1
Technical documents contain a fair amount of unnatural language, such as tables, formulas, pseudo-codes, etc. %While many efforts have been put for document layout analysis to identify figures, tables, formulas from PDF and OCR documents, but it has not been less explored from plain text.% Unnatural language can be an important factor of confusing existing NLP tools. This paper presents an effective method of distinguishing unnatural language from natural language, and evaluates the impact of unnatural language detection on NLP tasks such as document clustering. %As unnatural language blocks cause confusion for natural language processing (NLP), it is important to distinguish non-natural language blocks from natural language blocks. Previous work have been dedicated to identifying such blocks from PDF and OCR documents where visual layout information is preserved. %However, we ultimately deal with text for NLP, which motivated us to We view this problem as an information extraction task and build a multiclass classification model identifying unnatural language components into four categories. First, we create a new annotated corpus by collecting slides and papers in various formats, PPT, PDF, and HTML, where unnatural language components are annotated into four categories. We then explore features available from plain text to build a statistical model that can handle any format as long as it is converted into plain text. Our experiments show that removing unnatural language components gives an absolute improvement in document clustering up to 15\%. Our corpus and tool are publicly available. %Our model can handle any format as long as it can be converted into line-preserving plain text. %We show that removing unnatural language improves performance on an NLP task, specifically document clustering. %Through this paper, we introduce a new dataset of 70 computer science (34,200 lines) lecture slides and academic papers whose four types of unnatural language components are annotated. %In this paper, we study the automatic identification of the four types of non-natural language blocks, (1) Table (2) Code, (3) Mathematical Formula and (4) Miscellaneous in raw text extracted from technical documents. %We focus on documents in slide formats, which have been relatively under-explored. % %We view this problem as an information extraction task and build a multi-classification model trained on texts extracted from PDF, PPT, and HTML. We introduce a new dataset of 70 computer science (34,200 lines) lecture slides and academic papers whose four types of unnatural language components are annotated. %Our model can handle any format as long as it can be converted into line-preserving plain text. %We show that removing unnatural language improves performance on an NLP task, specifically document clustering.
naaclhlt2016.tex
%
% File naaclhlt2016.tex
%

\pdfoutput=1
\documentclass[11pt,letterpaper]{article}
\usepackage{naaclhlt2016}
\usepackage{times}
\usepackage{microtype}
\usepackage{latexsym}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{booktabs}
\usepackage{hyperref}

\usepackage{graphicx}
\usepackage{amsmath} 
\usepackage{algorithm2e}
%\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{color}
\usepackage{bold-extra}
\usepackage[T1]{fontenc}

\naaclfinalcopy % Uncomment this line for the final submission
%\def\naaclpaperid{***} %  Enter the naacl Paper ID here

% To expand the titlebox for more authors, uncomment
% below and set accordingly.
% \addtolength\titlebox{.5in}    

%\newcommand\BibTeX{B{\sc ib}\TeX}
\newcommand*{\myungha}{\textcolor{red}}
\setlength{\tabcolsep}{2pt}



%\title{A Study on Identification and Effects of Non-textual Components for Technical Documents Clustering}
%\title{Document Component Detection Without Layout \\ for Clustering Technical Documents} 
\title{Improving Document Clustering by Eliminating Unnatural Language}

% Author information can be set in various styles:
% For several authors from the same institution:
% \author{Author 1 \and ... \and Author n \\
%         Address line \\ ... \\ Address line}
% if the names do not fit well on one line use
%         Author 1 \\ {\bf Author 2} \\ ... \\ {\bf Author n} \\
% For authors from different institutions:
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \And  ... \And
%         Author n \\ Address line \\ ... \\ Address line}
% To start a seperate ``row'' of authors use \AND, as in
% \author{Author 1 \\ Address line \\  ... \\ Address line
%         \AND
%         Author 2 \\ Address line \\ ... \\ Address line \And
%         Author 3 \\ Address line \\ ... \\ Address line}
% If the title and author information does not fit in the area allocated,
% place \setlength\titlebox{<new height>} right after
% at the top, where <new height> can be something larger than 2.25in
\author{Myungha Jang$^{1}$, Jinho D. Choi$^{2}$, James Allan$^{1}$\\
	    $^{1}$College of Information and Computer Sciences, University of Massachusetts \\
	    $^{2}$Department of Computer Science, Emory University\\
	    {\tt mhjang@cs.umass.edu, jinho.choi@emory.edu, allan@cs.umass.edu}
}

\date{}

\begin{document}

\maketitle

\begin{abstract}
Technical documents contain a fair amount of \textit{unnatural language}, such as tables, formulas, pseudo-codes, etc. %While many efforts have been put for document layout analysis to identify figures, tables, formulas from PDF and OCR documents, but it has not been less explored from plain text.% 
Unnatural language can be an important factor of confusing existing NLP tools.
This paper presents an effective method of distinguishing unnatural language from natural language, and evaluates the impact of unnatural language detection on NLP tasks such as document clustering.
%As unnatural language blocks cause confusion for natural language processing (NLP), it is important to distinguish non-natural language blocks from natural language blocks. Previous work have been dedicated to identifying such blocks from PDF and OCR documents where visual layout information is preserved. %However, we ultimately deal with text for NLP, which motivated us to
We view this problem as an information extraction task and build a multiclass classification model identifying unnatural language components into four categories.
First, we create a new annotated corpus by collecting slides and papers in various formats, PPT, PDF, and HTML, where unnatural language components are annotated into four categories.
We then explore features available from plain text to build a statistical model that can handle any format as long as it is converted into plain text.
Our experiments show that removing unnatural language components gives an absolute improvement in document clustering up to 15\%.
Our corpus and tool are publicly available.
%Our model can handle any format as long as it can be converted into line-preserving plain text. 
%We show that removing unnatural language improves performance on an NLP task, specifically document clustering.
%Through this paper, we introduce a new dataset of 70 computer science (34,200 lines) lecture slides and academic papers whose four types of unnatural language components are annotated.

%In this paper, we study the automatic identification of the four types of non-natural language blocks, (1) Table (2) Code, (3) Mathematical Formula and (4) Miscellaneous in raw text extracted from technical documents. %We focus on documents in slide formats, which have been relatively under-explored. %
%We view this problem as an information extraction task and build a multi-classification model trained on texts extracted from PDF, PPT, and HTML. We introduce a new dataset of 70 computer science (34,200 lines) lecture slides and academic papers whose four types of unnatural language components are annotated. %Our model can handle any format as long as it can be converted into line-preserving plain text. 
%We show that removing unnatural language improves performance on an NLP task, specifically document clustering.  

\end{abstract}

\section{Introduction}

Technical documents typically include meta components such as figures, tables, mathematical formulas, and pseudo-code to effectively communicate complex ideas and results. Let us define the term \emph{unnatural language} as blocks of lines consist of only such components, as opposed to the body text that are natural language.


%that are not prose.

There are many great NLP tools available as the field has been advanced. However, %as the name \emph{natural language} processing obviously implies, %
these tools are mostly built for input text that are natural language. As many of our tools for NLP can be badly confused by unnatural language, it is necessary to distinguish unnatural language blocks from natural language blocks, or else unnatural language blocks will cause confusion for natural language processing. Once we salvage natural language blocks from the documents, we can exploit NLP tools much better as they are intended for. This phenomenon is emphasized in technical documents that have higher ratio of unnatural language compared to non-technical documents such as essays and novels. 




%Many of our tools for NLP can be badly confused by unnatural language. 





%\vspace{-0.3em}
Document layout analysis aiming to identify document format by classifying blocks into text, figures, tables, and such has been a long-studied problem \cite{gorman,simon}. 
Most previous work have focused on image-based documents, PDF and OCR formats, and used geometric analysis on the pages using the visual cues from its layout. This was a clearly important problem in many applications in NLP and IR.
%However, as our field advances, similar issues are becoming more and more important in other applications as well. 







%Technical documents frequently use non-textual components such as tables, mathematical formulas, pseudo-code, and diagrams for presenting ideas and results. 
%As the semantics of non-textual components come from their visual layout and logical flow, they are closer to \emph{noise} without humans' cognitive power to compile their meanings; for example, mathematical formulas are a sequence of numbers and arbitrary variable names. Each cell in a table can be understood only if its association with the row and the column is known \cite{tablesurvey}.

\begin{figure*}[t]
\centering
\includegraphics[scale=0.3]{img/similarcode2.PNG}
\caption{An example of how unnatural language confuses NLP tools. The left and right pseudo-codes are very different, but standard NLP similarity functions such as cosine similarity can easily be confused by the terms highlighted in yellow.}
\label{figure:similarcode}
\end{figure*}

%\vspace{-0.3em}

This work was particularly motivated while we attempted to cluster teaching documents (e.g., lecture slides and reading materials from courses) in technical topics. We discovered that unnatural language blocks introduced significant noise for clustering, causing spurious matches between documents. For example, code consists of reserved programming keywords and variable names. Two documents can contain two very different code from one another but their cosine similarity is computed high because they share many same terms by programming convention (Figure~\ref{figure:similarcode}). \cite{kohlhase} similarly recognized this problem by explaining main challenges of semantic search for mathematical formula: (1) Mathematical notation is context-dependent; without human's capability to understand the formula from the context, formulas are just \emph{noise}. (2) Identical presentations can stand for multiple distinct mathematical
objects. 


This paper proposes a new approach for identifying unnatural language blocks in plain text into four types of categories,(1) \textsc{table} (2) \textsc{code} (3) \textsc{mathematical formula}, and (4) \textsc{miscellaneous (misc.)}. Text are extracted from technical documents in the PDF, PPT, and HTML formats with little to no explicit visual layout information preserved. We focus on technical documents because they have a significant amount of unnatural language blocks (26.3\% and 16\% in our two corpora). Specifically, we focus on documents in slide formats, which have been relatively under-explored. %


%We investigate various features that are only available from plain text such as n-grams, string pattern, parsing structure, word embedding vectors, and sequential structure. Our approach is less format-dependent as it only requires lines to be preserved and does not depend on visual layout features that are only available from documents of certain file formats. Our evaluation shows that the model can successfully identify and classify unnatural language blocks into the four types, (1) \textsc{table} (2) \textsc{code} (3) \textsc{mathematical formula}, and (4) \textsc{miscellaneous (misc.)}, with F1-score higher than 80\%. 


We further study how removal of unnatural language improves NLP tasks, document similarity and document clustering.  Our experiments show that clustering on documents with unnatural language removed consistently showed higher accuracy on many of the settings than on original documents, with the maximum improvements up to 15\% and 11\% in two datasets, while it never significantly hurts the original clustering. 


%Another motivation of this work is that while identifying a certain component from documents has been studied, most of previous work was limited by the document file format that preserves visual layout, which are key features for component identification. Processing image-based documents (PPT, PDF, and OCR) is a costly operation when the original formats are available. Sometimes, there are no such \emph{original} formats available. Corpus from on-line technical forums or email thread is one example. Users often copy and paste their code error message. Clustering those documents can be challenging when the topic is often miss-classified by the the code than by the main text. 



%Many of our natural language processing tools can be badly confused by unnatural language: tables, formulas, pseudo-code, and etc. This paper proposes a new approach for distinguishing natural language from unnatural language, specifically focusing on the four components, (1) tables, (2) formulas, (3) pseudo-code and (4) miscellaneous text. We show that removing unnatural language improves performance on an NLP task, document clustering. 



%We demonstrate one application of our work  that treats the non-textual components as noise and removes them from the text as a document cleansing process. Our evaluation shows that F-score on clustering of component-removed documents was improved 11\% and 13\% respectively in two datasets compared to that of original documents in a simulated interactive setting. 









\section{Related Work}



\subsection{Table Extraction}
Various efforts have been made for table extraction using semi-supervised learning on the patterns of table layouts within ASCII text documents \cite{ng} web documents \cite{pinto,Lerman01automaticdata,survey} PDF and OCR image documents \cite{pdffigures,tableseer}. Existing techniques exploit the graphical features such as primitive geometry shapes, symbols, and lines to detect table borders. \cite{tablesurvey} introduces and compares the state-of-the-art table extraction techniques from PDF articles. However, there does not appear to be any work that has attempted to process plain text extracted from richer formats, where table layouts are unpreserved.

\subsection{Formula Extraction}
 \newcite{equInpdf} categorized existing approaches for mathematical formulas detection by `character-based' and `layout-based' with respect to key features.  \cite{mer} provides a comprehensive survey of mathematical formula extraction using various layout features available from image-based documents. Since we have no access to layout information, character-based approaches are more relevant to our work. They use features of mathematical symbols, operators, and positions and their character sizes \cite{suzuki,formulafuzzy}. %Our preliminary analysis suggests that character-based features are covered well by n-gram features. 
 
 
\begin{figure*}[t]
\centering
\includegraphics[scale=0.3]{img/table.png}
\caption{A table in PDF document (left) and its text-extracted version (right). Note that it is hard to distinguish the column headings from the extracted text without its layout.}
\label{figure:table-ex}
\end{figure*}




\begin{figure*}[t]
\centering
\includegraphics[scale=0.25]{img/misc.PNG}
\caption{Poor text extraction is worse than none. The output from Apache Tika (right) is useless (at best). Experiments will show that document clustering is improved by removing this kind of noise labeled as \textsc{Misc.}}
\label{figure:misc-ex}
\end{figure*}
 
 
 
\subsection{Code Extraction}
\newcite{tuarob} proposed 3 pseudo-code extraction methods: a rule based, a machine learning, and a combined method. Their rule based approach finds the presence of pseudo-code captions using keyword matching. The machine learning approach detects a box surrounding a sparse region and classifies whether the box is pseudo-code or not. They extracted four groups of features: font-style based, context based, content based, and structure based. 





\begin{table*}[t]
\centering
\begin{tabular}{@{}l|l|l@{}}
%\toprule
Purpose                                  & Name        & Content                    \\  \hline\hline
\multirow{3}{*}{Classification Training} & T$_{SLIDES}$   & 35 lecture slides (8,514 lines) whose components are annotated                                                        \\
                                         & T$_{ACL}$      & 35 ACL papers (25,686 lines) whose components are annotated    \\
                                         & T$_{COMBINED}$ & Combination of T$_{SLIDES}$ and T$_{ACL}$ \\
                                          \hline
Word Embedding Training                  & T$_{WORD2VEC}$ & \begin{tabular}[c]{@{}l@{}}1,190 lecture slides and 5,863 ACL/EMNLP papers archived \\ over a few years that are used for training word embedding.\end{tabular} \\ \hline
\multirow{2}{*}{Clustering}              & C$_{DSA}$      & 128 lecture slides from `data structure' and `algorithm' classes                                                    \\
                                         & C$_{OS}$       & 300 lecture slides from, `operating system' classes                                                                                                             \\
\end{tabular}
\caption{Datasets used in our paper. All data are available for download at [\url{http://people.cs.umass.edu/~mhjang/publication.html}]}
\label{tab:data-summary}
\end{table*}




\section{Problem Definition}
Input to our task is the plain text extracted from PDF or PPT documents. The goal is to assign a class label to each line in that plain text, identifying it as natural language (regular text) or one of the four types of non-natural language block components, table, code, formula, or miscellaneous text. 
In this work, we focus on these four specific types because our observations lead us to believe they are the most frequently occurring components in PPT lecture slides and PDF articles. Figures are also a very frequent component but we do not consider them because they are commonly pictures or drawings and cannot be easily extracted to text. In this section, we briefly discuss the characteristics of each component and challenges in their identification from the raw text. 

\subsection{Table}
Tables are prevalent in almost every domain of technical documents. Tables are usually conveyed by its two-dimensional layout and its column and/or row headings \cite{tablesurvey}. Figure \ref{figure:table-ex} shows a table in an original PDF document and the same table as it appears the text extracted by Apache Tika\footnote{https://tika.apache.org/}. Tables frequently have multiple cells merged for layout, which makes them particularly difficult to distinguish as table once they are converted to flat text.  




\subsection{Mathematical Formula}
Mathematical formulas exist in two ways: isolated formulas on their own lines or as formulas embedded within a line of text. In this work, we treat both types as formula component. Because not all math symbols can be matched to Unicode characters and because the extraction software may not convert them correctly, the extracted text tends to contain more oddly formatted or even completely wrong characters. Superscripts and subscripts are no longer distinguishable and the original visual layout (e.g., math symbols over multiple lines such as $\Pi$ and $\sum$) is lost. 

\subsection{Code}
Articles in Computer Science or related fields often contain pseudo-code or actual program code to illustrate their algorithm. Similar to mathematical formulas, they exist both isolated and embedded, though most code components are isolated code blocks. As in formula component, we treat both types of code blocks as code components. We assume that even indents, one of the strong code visual cues, are not preserved in the extracted text although some extraction tool saves them, not to limit ourselves to the detailed performances of text extraction tools.

\subsection{Miscellaneous Non-text (Misc.)}
In addition to the components mentioned above, there are other types of non-natural language blocks that are left during conversion to text and that may provide spurious sub-topic matches between documents. To allow for those, we denote those components as miscellaneous text. One example of miscellaneous text is the text and caption that are part of the diagrams in slides. Figure \ref{figure:misc-ex} shows an example of miscellaneous text that lost its structure and meaning while being converted to text without the original diagram.  






\section{Corpus}
\subsection{Data Collection}
We collected 1,561 lecture slides from various Computer Science and Electrical Engineering courses that are available online, and 5,898 academic papers from several years of ACL/EMNLP archive\footnote{https://aclweb.org/anthology}. We divided the dataset for several purposes; training the classification model, training word embedding model for feature extraction, and clustering for extrinsic evaluation. The details of the dataset we used are summarized in Table ~\ref{tab:data-summary}. We make the data publicly available for download at \url{http://people.cs.umass.edu/~mhjang/publications.html}. 





For classification, we constructed three dataset using two different data sources: (1) lecture slides (2) ACL papers, (3) combining both. We chose these two types of data sources because they have different ratios of unnatural language components, hence complementary to each other for the coverage. Table 2 shows the ratio of the four components from each annotated dataset. For example, 1.4\% of lines in $T_{SLIDES}$ are annotated as part of table. 





\subsection{Text Extraction}
We extracted plain text from our datasets using an open-source software package, Apache Tika. The package is available for text extraction from various formats including PDF, PPT, and HTML.

\subsection{Annotation}
To train a statistical model, we need ground-truth data. We created annotation guidelines for the 4 types of non-natural language components and annotated 35 lectures slides (7,943 lines) and 35 ACL proceeding papers (25,686 lines). We developed a simple annotation tool to support the task and also to enforce that annotators follow certain rules\footnote{The guidelines and the tool are available at \url{http://people.cs.umass.edu/~mhjang/publication.html}}. We hired four undergraduate annotators who have knowledge of the Computer Science domain for this task. 



\setlength{\tabcolsep}{1pt}



\begin{table}[ht]
\label{tab:component_ratio}
\centering
\footnotesize
\begin{tabular}{cccccc}

     & \multicolumn{1}{c}{\textsc{Table}} & \multicolumn{1}{c}{\textsc{Code}} & \multicolumn{1}{c}{\textsc{Formula}} & \multicolumn{1}{c}{\textsc{Misc.}} & 
     \begin{tabular}[c]{@{}l@{}}\scriptsize{All Unnatural}\\ \scriptsize{Categories}\end{tabular}
      \\ \hline
\multicolumn{1}{c}{T$_{SLIDES}$} & 1.4\%  & 14.6\%  & 0.5\%  & 9.8\% & 26.3\% \\ \hline
\multicolumn{1}{c}{T$_{ACL}$}    & 4.0\%  & 0.6\%   & 5.0\%  & 6.4\% & 16\% \\ \hline          
\end{tabular}
\caption{\% of lines by unnatural category. Both datasets have quite a bit of unnatural language (26.3\% for T$_{SLIDES}$ and 16\% for T$_{ACL}$), though T$_{ACL}$ has more \textsc{tables} and \textsc{formulas} and less \textsc{code}.
}
\label{my-label}
\end{table}


%The general guidelines are as following: 
%\begin{itemize}
%\item Identify each component and add the begin tag and the end tag on the beginning and the ending of it using the eight tag buttons. If the tag is located in the beginning of the line, click the begin tag button with the select cursor on the left list. If the tag should be located in the middle of the sentence, use the right list that provides a word level view.
%\item Tags cannot be nested. For example, \textless TABLE\textgreater \textless CODE \textgreater \textless/CODE\textgreater\textless /TABLE\textgreater is not allowed.
%\item We assume that there is only one type of component in one line. \item Hence, one type of tag can be located in one line. If there are multiple, we choose the one that has a longer length. 
%\item Once a begin tag is located, there has to be the matching end tag.
%\end{itemize}



\section{Features}
We find line-based prediction has more advantage over token-based prediction because it allows us to observe the syntactic structure of the line, how statistically common the grammar structure is, and how layout patterns compare to neighboring lines. We introduce five sets of features used to train our classifier and discuss each feature's impact on the accuracy.

\subsection{N-gram (N)} Unigrams and bigrams of each line are included as features. 

\subsection{Parsing Features (P)} Unnatural languages are not liklely to form any grammar structure. When we attempt to parse the unnatural language line, the resultant parsing tree would form unusual syntactic structure. To capture this insight, we parsed each line using the dependency parser in ClearNLP \cite{choi:13a} and extracted features such as the set of dependency labels, the ratio of each POS tag, and POS tags of each dependent-head pair from each parse tree. 

\subsection{Table String Layout (T)} Text extracted from tables loses its visual layout as a table but still preserves implicit layout through its string patterns. Tables tend to convey the same type of data along the same column or row. For example, if a column in a table reports numbers, it is more likely to contain numeral tokens in the same location of the lines of the table in parallel. Hence, a block of lines will more likely be a table if they share the same pattern. We encode each line by replacing each token as either S (String) or N (Numeral). We then compute the edit distance among neighboring lines weighted by language modeling probability computed from table corpus (Equation \ref{table-equ}, \ref{table-lm}). 
\begin{multline}
\label{table-equ}
 P_{table}(l_{i}) \propto P_{table}(l_{i} | l_{i-1}) 
\\ = TableLanguageModel(l_{i}) \cdot  \\ editDistance(encode(l_{i}), encode(l_{i-1}))
\end{multline}
\vspace{-4em}

\begin{multline}
\label{table-lm}
  TableLanguageModel(l_i) \\
  = \Pi_{j}^{n}{(P(encode(t_{i,j+1}) | encode(t_{i,j}))} 
\end{multline}
  where $l_i$ refers to a i-th line in a document, $t_{i,j}$ refers to a j-th token in $l_i$.


%A single token is not always mapped to each cell; there can be more than one word in a cell or merged cell for a certain layout. In this case, we noticed that the table shows a better pattern when aligning tokens leftwards from the right end because the pattern of the left-side of the table is typically less obvious with heading and merge-layout. We use a feature of the ratio of the consecutive exact match between the current and previous encoded lines leftwards and rightwards1.

\subsection{Word Embedding Feature (E)} We train word embeddings using $T_{WORD2VEC}$ using \textsc{word2vec} \cite{embedding}. The training corpus contained 278,719 words. Since we do a line-based prediction, we need a vector that represents the line, not each word. We consider three ways of computing line embedding vector: (1) by averaging the vector of the words, (2) by computing paragraph vector introduced in \cite{le}, (3) using both.

\subsection{Sequential Feature (S)}  The sequential nature of the lines is also an important feature because the component most likely occurs over a block of contiguous lines. We train two models. The first model uses the annotation for the previous line's class. We then train another model using the previous line's predicted label, which is the output of the first model.



\setlength{\tabcolsep}{4pt}


\begin{table*}[ht!]
\centering
\begin{tabular}{c|cccc|cccc}
                    &\multicolumn{4}{c|}{T$_{SLIDES}$}                            & \multicolumn{4}{c}{T$_{ACL}$}                               \\ \hline\hline
                    & \textsc{table} & \textsc{code} & \textsc{formula} & \textsc{misc.} & 
\textsc{table} & \textsc{code} & 
\textsc{formula} & 
\textsc{misc.} \\
%\textbf{Majority}   & 0.00           & 0.00          & 0.00             & 0.00          & 0.00           & 0.00          & 0.00             & 0.00          \\
W-Random   & 1.69           & 14.62         & 2.82             & 10.57         & 4.15           & 0.62          & 4.44             & 6.08          \\
CLM         & 5.41           & 28.62         & 0.00             & 10.47         & 13.10          & 16.45         & 10.32            & 5.18          \\
Proposed Method & \textbf{67.89}          & \textbf{90.22}         & \textbf{29.09}            & \textbf{89.63}         & \textbf{86.58}          & \textbf{63.70}         & \textbf{80.98}            & \textbf{87.63}        \\
\hline\hline
PC-CB \cite{tuarob}* & N/A & 75.95 & N/A & N/A & N/A & 75.95 & N/A & N/A \\
\end{tabular}
\caption{Single-domain Classification Result in F1-score: Proposed method is much better than baselines for classifying unnatural language. *Note that we borrowed the F1-score reported on their dataset for reference. The number is not directly comparable to other numbers since the datasets are different.}
\label{result:singledomain}
\end{table*}

\section{Classification Experiments}
We used the Liblinear Support Vector Machine (SVM) \cite{svm} classifier for training and ran 5-fold cross-validation for evaluation. To improve the robustness of structured prediction, we adopted a learning to search algorithm known as \textsc{DAgger} to SVM \cite{dagger}.  We first introduce two baselines to compare the accuracy against our statistical model. %We then present classification result of our model, %and report the comparison of clustering results before and after removing non-natural language blocks. 

\subsection{Baselines} 
Since no existing work is directly applicable to our scenario, we consider two straightforward baselines.
\begin{itemize} 
%\item \textbf{{Majority Class (Majority)}} Assign the majority-class component to each line. However, for both of the dataset, the majority class is always TEXT because the ratio of the non-textual components take up only 26.5\% and 20\% respectively for TSLIDES and TACL. The accuracy for the four components classification task is 0 because this makes a safe guess that a given line is never any component. 
\item \textbf{Weighted Random (W-Random)} \\ This assigns the random component class to each line. Instead of uniform random prediction, we made more educated guesses using the ratio of components known from the annotated dataset (Table 2). %For example, each line has a 14.6\% chance of being CODE in $T_{SLIDES}$. 

\item \textbf{Component Language Modeling (CLM)} \\ Among the five language models of five component class (the four non-textual components  and text component) generated from the annotations, we predict the component for each line by assigning the component whose language model gives the highest probability to the line. %Since models have varying amount of data, we also experimented to find the optimal smoothing parameters to mix with the collection probability when the word is unseen from the model using 5-fold cross validation. %Our experiment showed that the models that did not use any smoothing performed the best. The smoothing with the collection probability hurt the precision more than the recall gained. Therefore, we did not use any smoothing in our experiments. 
\end{itemize}

\subsection{Classification Result} 
We first conducted single-domain classification. Annotations within each dataset, T$_{SLIDES}$ and T$_{ACL}$ are split for training and testing using 5-fold cross validation scheme. Table \ref{result:singledomain} reports F1-score for prediction of the four components in the two dataset using our method as well as baselines.  




\begin{table}[h]
\centering
\begin{tabular}{@{}cccc@{}}

\toprule
        & Precision & Recall & F1-score \\ \midrule
Table   & 94.60     & 76.39  & \textbf{84.53}      \\
Code    & 89.56     & 84.01  & \textbf{86.69}      \\
Formula & 85.07     & 79.32  & \textbf{82.10}      \\
Misc.   & 85.59     & 90.24  & \textbf{87.86}      \\ 
Text & 97.76 & 98.79 & 98.27 \\
\bottomrule
\end{tabular}
\caption{Multi-domain classification improves the single-domain classification in Table \ref{result:singledomain}. Identification of categories with particularly low accuracy in each datasets (\textsc{table} and \textsc{formula} in T$_{SLIDES}$ and \textsc{code} in T$_{ACL}$) are improved to be as good as the other categories. }
\label{cross-domain}
\end{table}


Proposed method dramatically increased the prediction accuracies for all of the components against the baselines. CLM baseline showed the highest accuracy on \textsc{code} among the four categories in both datasets. Because pseudo-codes use more controlled vocabulary (e.g., reserved words, common variable names), the language itself becomes distinctive characteristics. We also include the numbers reported by \newcite{tuarob} for comparison. Since their dataset was 258 PDF scholarly articles, T$_{ACL}$ is more comparable dataset than T$_{SLIDES}$, but our training set is much smaller than their dataset. However, their number reported on Table 3 is not directly comparable to other numbers because the numbers are on different datasets.


In T$_{SLIDES}$, the classification F1-score for \textsc{formula} is relatively low as 29.09\% compared to the other components in the same dataset, and also compared to the \textsc{formula} prediction in T$_{ACL}$ (80.98\%). This is due to too small amount of training data (only 0.5\% of \textsc{formula} in T$_{ACL}$), which is overcome in T$_{SLIDES}$ that contain 5\% of \textsc{formula} training data (refer to Table 2).  %Although only 0.6\% of \textsc{code}  is used in T$_{ACL}$, the size of T$_{ACL}$ is three times bigger than T$_{SLIDES}$.%


In the proposed method, classification of \textsc{code} and \textsc{misc} was significantly improved in T$_{SLIDES}$ (around 90\%), while that of \textsc{table} and \textsc{formula} was improved in T$_{ACL}$ (over 80\%). This shows the complementary nature between two datasets, which suggests that a combined dataset of two, T$_{combined}$, would improve classification performance. Table 4 shows the multi-domain classification result using T$_{combined}$, in which all four categories are identified with F1-score higher than 80\%. 









\subsection{Feature Analysis}
We conducted feature analysis to understand the impact of single feature and their combination. We started from single features and incrementally combined them to observe the performance (Figure \ref{figure:figure_analysis}). Features are added in a greedy fashion that a feature that gives the higher accuracy when used alone is added first. 



\begin{figure}[t]
\centering
\includegraphics[scale=0.35]{img/embedding.png}
\vspace{-3mm}
\caption{Three ways of computing sentence embedding vector}
\label{figure:embedding}
\end{figure}


\begin{figure}[h]
\centering
\includegraphics[scale=0.35]{img/feature_analysis.png}
\vspace{-3mm}
\caption{Feature analysis for \textsc{table} and \textsc{formula}. N: N-gram, E: Embedding, P: Parsing, T: Table String Layout, S: Sequential.  identification in T$_{combined}$}
\label{figure:figure_analysis}
\end{figure}







We first compare the three ways of computing sentence vector features mentioned in Section 5 (Figure \ref{figure:embedding}). When we experiment with only embedding features, averaging word vectors performed 9-12 times better than paragraph vectors. When both features were used, there are some gains in \textsc{Code} and \textsc{Misc.} components and losses in \textsc{Table} and \textsc{Formula}. However, when we experiment with all the other features in addition to embedding features, losses were covered by the other features such that ultimately combined vectors give overall the highest performances. 


N-gram (N) features was the most powerful feature with 68\% of F1-score when used alone. The next useful features are parsing feature (P), table layout (T), and embedding features (E) in order for \textsc{table}, while embedding vectors were more effective than parsing feature for \textsc{code} (Figure 5). 

%The sequential features cannot be observed alone because it depends on the first prediction made on the previous line. 

%are almost as twice as stronger evidence for most of the components than sentence embedding features alone. Parsing feature gave a boost in Table identification in addition to N-gram and embedding features. 

%The model performed much better with N-gram features than word-embedding vectors. Word embedding vectors brought 1-2\% increases when they are added to N-gram features. 
%Parsing features was most effective on TABLE identification, which increased the accuracy by 9.3\% when they are added to N-gram + Embedding features. Table layout feature was also most effective for TABLE identification, which brought 4.2\% gain, but also improved EQUATION identification by 3.6\%, which suggests that EQUATION components also tend to have the similar string/numerical patterns. 

%Lastly, sequential feature was added. MISC component gained almost 6\% and EQUATION gained 3\%. 

%We have shown that the five set of features as a combination can identify the four types of components from the text lines with above 80\% F-score. As our motivation to identify the components was to study its effects to clustering, we take the trained model and observe how identification of components with 82-87\% F-score affects the clustering performance. 




\section{Removal Effects of Unnatural Language on NLP tools} 
We observe how removal of unnatural language from document affects the performance of two NLP tools, document similarity and document clustering. For the set of experiments, we prepared a gold standard clustering for each dataset, C$_{DSA}$ and C$_{OS}$. 


%We want to find an answer to two research questions:

%\emph{Q1: How does removal of unnatural language blocks affect the document topic representation?}

%\emph{Q2: How does removal of unnatural language blocks affect the document clustering?}


%Note that the two questions address the two different issues. Improvement of documents' topic representation might not always necessarily improve clustering. To study the first question, we run regression analysis on the two sets of documents, with and without non-textual components. 

%For the next experiments, we prepared a topic list and a gold standard clustering for the topic list for two slides dataset, C$_{DSA}$ and C$_{OS}$. 

 %We hypothesized that these non-textual components reduced clustering quality so that removal of them would improve the document representation and hence clustering result. To verify this, we prepared two sets of gold standard for C$_{DSA}$ and C$_{OS}$ (see Table \ref{table:data-summary}).
 
 % For each dataset, we prepared following:

%\begin{itemize}
%\item topic list (APPENDIX A, B) 
%\item Gold standard document clustering by the topic list 
%\end{itemize}

%This is a tricky evaluation because the clustering performance varies by many factors, e.g., document representation, similarity function, clustering algorithms. Here we adopted the most standard way of document representation; representing document as a bag-of-words with their TF/IDF score. As one way to evaluate whether the document representation is improved after the non-textual components are removed, without the effects of clustering algorithm, we first conducted regression analysis. 

\subsection{Document Similarity}
If two documents are similar, they must be topically relevant to each other. A good similarity measure should reflect that; two topically relevant documents should have a high similarity score. To test whether the computed similarity reflects the actual topic relevance better once the unnatural language is removed, we conducted regression analysis.

We converted the gold standard clustering to pair-wise binary relevance. If two documents are in the same ground-truth cluster, they are relevant, and otherwise irrelevant. We then fitted a log-linear model in R for predicting binary relevance from the cosine similarity of document pairs. 

Regression models fitted in R are evaluated using AIC \cite{akaike1974}. The AIC is a measure used as a means for model selection, which measures the relative quality of statistical models learned from the given data. When AIC is smaller, the goodness of fit is better, and the smaller the complexity of the model is, having fewer parameters to represent the model. Table \ref{tab:regression} shows that AIC was reduced by 53 and 118 respectively on the models trained with documents whose unnatural language blocks are removed, compared to the original documents. Since AIC does not provide a test for a model, AIC does not suggest anything about the quality of the model in an absolute sense, but relative quality. From this result, we can conclude that cosine similarity can fit a better model that predicts documents' topic relevance with significance after unnatural language blocks have been removed. 


%\setlength{\tabcolsep}{2pt}

\begin{table}[h]
\centering
\small
\begin{tabular}{c|c|c|c}

    & AIC(D$_{original}$) & AIC(D$_{removed}$) & Improvement \\ \hline
C$_{DSA}$ &   -40975   &     -41028    & -53             \\
C$_{OS}$  &    -61404  & -61522    & -118           
\end{tabular}
\caption{
The statistical model is trained better with documents whose unnatural language categories are removed (D$_removed$) than the model with the original documents (D$_original$) in both datasets.  \emph{Smaller} AIC scores imply \emph{better} models.}
\label{tab:regression}
\end{table}

\begin{figure}[t]
\centering
\includegraphics[scale=0.25]{img/clustering_big_tight.png}
\caption{Clustering result on two datasets, C$_{DSA}$ (top) and C$_{OS}$ (bottom). X axis referes to the the size of document vector K, which controls the top-K TF-IDF terms included from documents. Y axis: Clustering F1-score. }
\label{figure:clustering}
\end{figure}







%\begin{figure*}[ph!]
%\centering
%\includegraphics[scale=0.4]{img/dsa_clustering.png}
%\caption{The trend of classification accuracy changes as a set of features are incrementally added.}
%\label{figure:figure_analysis}
%\end{figure*}
  
  \vspace{-2em}

\subsection{Document Clustering} 
Comparing general clustering performance on two document sets is tricky because clustering performance varies by many factors, e.g., clustering algorithm, similarity function, document representation, and parameters. To make a safe claim that clustering quality of one set of documents is better than of the other, clustering on one set should consistently outperform the other under many different settings. To validate this, we perform clustering experiments with multiple settings such as varying document vector size and and different initialization schemes. 



In this experiment, we consider seeded K-means clustering algorithm \cite{kmeans} for teaching documents. In our application scenario, users initially submit a topic list (e.g., syllabus) of the course. Then lecture slides are grouped into the given topic cluster. Depending on users' interaction level, we consider a semi-interactive scenario where users only provide a topic list, and a fully-interactive setting where users not only provide a topic list but also provide an answer document for each topic cluster, more specifying the intended topic. 






  



\begin{algorithm}[ht]
\label{seeded_kmeans}
 \KwData{Set of document vectors D = \{$d_1, ... d_n$\}, $d_i \in R^T$, set of seed vectors S = \{$s_1, ...s_k$\}, user-provided topic keywords vector T = $\{t_1,...t_k\}$}  \KwResult{Disjoint K partitioning of D into C$_{l=1}^{k}$}
 
 {Seed Initialization:}\\
 \phantom{haa}(1) Topic-keywords seeding:    $s_i = t_i$,  \\
 \phantom{haa}(2) Top-1 document seeding: \\  
 \phantom{haaa}$s_i = d_j$  $argmax_j(\textsc{cosineSimilarity}(t_i, d_j))$ \\
 \phantom{haa}(3) User-selected document seeding: \\  
 \phantom{haaa}$s_i = $\textsc{DocselectedByUser}($t_i)$  \\
 \While{convergence}{
  K-means clustering document selection process
 }
 \caption{Seeded K-means with User Interaction}
\end{algorithm}


In a semi-interactive setting, topic keywords are sparse seed as they usually consist of two or three words. Therefore, we expand the topic keywords by finding the top-1 document retrieved from the keywords and use it as seed. For experiments, we simulate the fully-interactive setting; instead of having an actual user to pick an answer document, we use an answer document randomly chosen from a gold cluster. The seeded K-means clustering algorithm with three interactive seeding schemes is described in Algorithm 1. 




%        DSA   OS   Ratio(DSA) Ratio(OS) 
%Top 10	  2	   9	13%	        60%
%Top 20	  6	   11	40%	        73%
%Top 30	  9	   13	60%	        86%
%> 	      15   15	100	        100%










We can consider a simulated setting more realistic when the selected document is suggested to the user as the top or a near-top choice. In our dataset, 60\% of the selected documents were ranked in top 10 %and 74\% in top 20 %
in C$_{DSA}$, and 13\% of the selected documents were ranked in top 10 in C$_{OS}$, %and 40\% in top 20, %
which implies that the simulated setting in C$_{DSA}$ was more realistic than in C$_{DSA}$. For top-1 document seeding, 64\% and 78\% of document seeds matched with the gold standard in C$_{DSA}$ and C$_{OS}$, respectively. 

Figure \ref{figure:clustering} shows the clustering result of original documents (D$_{original}$) and documents whose unnatural language blocks removed (D$_{removed}$), with three different seeding schemes over two lecture slides datasets. In C$_{DSA}$, D$_{removed}$ consistently outperformed with all three seeding schemes. The clustering performed the best with D$_{removed}$ when top-1 document was used as seed. Overall, in C$_{DSA}$, clustering was improved 94\% of the times with the maximum absolute gain of 14.7\% and the average absolute gain of 4.6\%. The average absolute loss was 0.8\% when 6\% of the times it hurt. In C$_{OS}$, clustering was improved 73\% of the times with the maximum absolute gain of 11.4\% and the average absolute gain of 3.9\%. The average absolute loss was 1.7\%. Our results suggest that removal of unnatural language blocks can significantly improve clustering most of the times with bigger gain than occasional losses. 


%In both datasets, clustering on components-removed documents consistently showed improvements with an answer document is given as cluster seed for all clusters. When the documents automatically found from topic keywords were used as seed, (i.e., clusters partially had the answer seeds), two datasets showed different patterns. 



%For clustering, we considered a seeded K-means clustering [15] because unlike an unsupervised clustering scenario that the number of topics or the topic category is unknown, our problem of clustering class lecture slides almost always has an access to the knowledge of the topic category, such as from textbook’s table of contents or lecture syllabus. The seeded K-means clustering initializes the centroids with the given context (Table 6). 

%However, the seeded clustering with the topic keywords (Appendix A&B) are not stable. Because each cluster’s initial topic keywords consisted of 2-3 terms on average, the results often dramatically fluctuated how topic keywords are chosen. With a little change of wording or vocabulary often gives depending on the different clustering result as it pulls different sets of documents at the initial stage. To obtain more stable clustering result, we considered two other variant of seed selection in addition to topic-keywords.

%Topic keyword seed (KS)
%Document Seed Closest to the topic keywords (DS): This selects the most-similar document to the given topic keywords and use the document as seed	
%Answer document seed (AS): we randomly assign a gold document as seed for each cluster instead of topic keywords. Note that this is an oracle experiment introduced to simulate a user interaction in which the system shows a list of relevant page candidates for each topic and a user can manually select one to ask the system ‘find me more like this document’. 
%The size of the document vector also affects the clustering performance. A naïve way would be to use all words in a document to construct the vector. Alternatively, instead of using all words, we can select important keywords. To weigh the term importance, we use TF/IDF score. We set a parameter K, which controls the maximum number of terms extracted from each document. For example, K=30, a document vector consists of top 30 terms with high TF/IDF score. For term candidates, we consider unigrams and bigrams of the tokenized and stemmed words in a document.   
  
  


  
  
%Clustering consistently performed better with all of three initializations in CDSA (Figure 8 and 9). This clearly shows that the documents can be better clustered after the components are removed in CDSA. We have more conflicting results in COS. Clustering with keywords seeding on documents whose components are removed performed slightly less when the document vector size is small, but eventually it catches up with the other setting. The first two graphs in Figure 9 seems to suggest that components removal on COS does not have positive effect on clustering. 

%The issue with topic-keywords or topic-keywords induced seeding is that since the initial keywords are sparse, they provide less accurate context, and often the errors can be propagated when finding the seeding document. Table 7 shows an example of initial centroids using three seeds used for initialization. We can see that the terms selected by answer document seeding contains more relevant terms at higher rank, such as core l1, worker thread, TCB, pthread, than the terms selected by keywords-induced document seeding, which was selected based on only one topic term, thread. 
%     When we used an answer document seed, the result showed a completely different pattern than the first two results in Figure 9. In both datasets, clustering of components-removed documents was significantly improved in all kinds of size of a document vector except for one case. Based on the results, we can conclude that the identification and removal of components on the documents had an impact that clustering was improved by 11\% and 13\% respectively in the two datasets in especially an interactive setting.   


\vspace{-0.5em}
\section{Conclusion}

In this paper, we argued that unnatural language should be distinguished from natural language in technical documents for NLP tools to work effectively. We presented an approach to the identification of four types of unnatural language blocks from plain text, which is not dependent a document format. Proposed method extracts five sets of line-based textual features, and showed above 82\% F1-score for the four categories of unnatural language.  
We showed how existing NLP tools can work better on documents if we remove unnatural language from documents. Specifically, we demonstrated removing unnatural language improved document clustering in many settings by up to 15\% and 11\% at best, while not significantly hurting the original clustering in any setting. 

%We suggest two lines of future work from this. As we assume the line-based prediction in our model, we assumed that each line is always part of a component as a whole when it is. However, this assumption is not always true. A component, such as formula, can be embedded within a sentence. To make more accurate prediction, prediction should be made at a token-level. 

%The future work is to understand the semantics of the identified components. Since some parts of the components can still be relevant to the topic of the document, we might have lost some relevant information by treating all components as noise. %For example, table’s cell values might not be useful, but table’s metadata, column/row names, can still contain useful information. 
%Recovering the semantics of the components will also be useful for enabling component-aware document search, queries such as \textit {documents with DFS pseudo-code}, or re-ranking the documents by the ratio of the components. 






\section*{Acknowledgments}
This work was supported in part by the Center for Intelligent Information Retrieval, in part by NSF grant \#IIS-0910884, and in part by NSF grant \#IIS-1217281. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor. The authors thank Kenneth W. Church for providing valuable comments and advice. 

%\clearpage
\bibliography{naaclhlt2016}
\bibliographystyle{naaclhlt2016}


\end{document}

naaclhlt2016.bbl
\begin{thebibliography}{}

\bibitem[\protect\citename{Akaike}1974]{akaike1974}
Hirotugu Akaike.
\newblock 1974.
\newblock {A new look at the statistical model identification}.
\newblock {\em Automatic Control, IEEE Transactions on}, 19(6):716--723,
  December.

\bibitem[\protect\citename{Basu \bgroup et al.\egroup }2002]{kmeans}
Sugato Basu, Arindam Banerjee, and Raymond~J. Mooney.
\newblock 2002.
\newblock Semi-supervised clustering by seeding.
\newblock In {\em Proceedings of the Nineteenth International Conference on
  Machine Learning}, ICML '02, pages 27--34, San Francisco, CA, USA. Morgan
  Kaufmann Publishers Inc.

\bibitem[\protect\citename{Chan and Yeung}2000]{mer}
Kam-Fai Chan and Dit-Yan Yeung.
\newblock 2000.
\newblock Mathematical expression recognition: A survey.

\bibitem[\protect\citename{Chang and Lin}2011]{svm}
Chih-Chung Chang and Chih-Jen Lin.
\newblock 2011.
\newblock {LIBSVM}: A library for support vector machines.
\newblock {\em ACM Transactions on Intelligent Systems and Technology},
  2:27:1--27:27.

\bibitem[\protect\citename{Choi and McCallum}2013]{choi:13a}
Jinho~D. Choi and Andrew McCallum.
\newblock 2013.
\newblock Transition-based dependency parsing with selectional branching.
\newblock In {\em Proceedings of the 51st Annual Meeting of the Association for
  Computational Linguistics}, ACL'13, pages 1052--1062.

\bibitem[\protect\citename{Clark and Divvala}2015]{pdffigures}
Christopher Clark and Santosh Divvala.
\newblock 2015.
\newblock Looking beyond text: Extracting figures, tables and captions from
  computer science papers.
\newblock In {\em AAAI Workshops}.

\bibitem[\protect\citename{Kacem \bgroup et al.\egroup }2001]{formulafuzzy}
Afef Kacem, Abdel Bela{\"\i}d, and Mohamed~Ben Ahmed.
\newblock 2001.
\newblock Automatic extraction of printed mathematical formulas using fuzzy
  logic and propagation of context.
\newblock {\em IJDAR}, 4(2):97--108.

\bibitem[\protect\citename{Khusro \bgroup et al.\egroup }2015]{tablesurvey}
Shah Khusro, Asima Latif, and Irfan Ullah.
\newblock 2015.
\newblock On methods and tools of table detection, extraction and annotation in
  pdf documents.
\newblock {\em J. Inf. Sci.}, 41(1):41--57, February.

\bibitem[\protect\citename{Kohlhase and Sucan}2006]{kohlhase}
Michael Kohlhase and Ioan Sucan.
\newblock 2006.
\newblock A search engine for mathematical formulae.
\newblock In Jacques Calmet, Tetsuo Ida, and Dongming Wang, editors, {\em
  AISC}, volume 4120 of {\em Lecture Notes in Computer Science}, pages
  241--253. Springer.

\bibitem[\protect\citename{Le and Mikolov}2014]{le}
Quoc~V. Le and Tomas Mikolov.
\newblock 2014.
\newblock Distributed representations of sentences and documents.
\newblock {\em CoRR}, abs/1405.4053.

\bibitem[\protect\citename{Lerman \bgroup et al.\egroup
  }2001]{Lerman01automaticdata}
Kristina Lerman, Craig Knoblock, and Steven Minton.
\newblock 2001.
\newblock Automatic data extraction from lists and tables in web sources.
\newblock In {\em In Proceedings of the workshop on Advances in Text Extraction
  and Mining (IJCAI-2001), Menlo Park}. AAAI Press.

\bibitem[\protect\citename{Lin \bgroup et al.\egroup }2011]{equInpdf}
Xiaoyan Lin, Liangcai Gao, Zhi Tang, Xiaofan Lin, and Xuan Hu.
\newblock 2011.
\newblock {Mathematical Formula Identification in PDF Documents}.
\newblock In {\em International Conference on Document Analysis and
  Recognition}, ICDAR, pages 1419--1423.

\bibitem[\protect\citename{Liu \bgroup et al.\egroup }2007]{tableseer}
Ying Liu, Kun Bai, Prasenjit Mitra, and C.~Lee Giles.
\newblock 2007.
\newblock {TableSeer: automatic table metadata extraction and searching in
  digital libraries}.
\newblock In {\em Joint Conference on Digital Library}, JCDL, pages 91--100.

\bibitem[\protect\citename{Mikolov \bgroup et al.\egroup }2013]{embedding}
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.
\newblock 2013.
\newblock Efficient estimation of word representations in vector space.
\newblock {\em CoRR}, abs/1301.3781.

\bibitem[\protect\citename{Ng \bgroup et al.\egroup }1999]{ng}
Hwee~Tou Ng, Chung~Yong Lim, and Jessica Li~Teng Koo.
\newblock 1999.
\newblock Learning to recognize tables in free text.
\newblock In {\em Proceedings of the 37th Annual Meeting of the Association for
  Computational Linguistics on Computational Linguistics}, ACL '99, pages
  443--450, Stroudsburg, PA, USA. Association for Computational Linguistics.

\bibitem[\protect\citename{O'Gorman}1993]{gorman}
L.~O'Gorman.
\newblock 1993.
\newblock The document spectrum for page layout analysis.
\newblock {\em IEEE Trans. Pattern Anal. Mach. Intell.}, 15(11):1162--1173,
  November.

\bibitem[\protect\citename{Pinto \bgroup et al.\egroup }2003]{pinto}
David Pinto, Andrew McCallum, Xing Wei, and W.~Bruce Croft.
\newblock 2003.
\newblock Table extraction using conditional random fields.
\newblock In {\em Proceedings of the 26th Annual International ACM SIGIR
  Conference on Research and Development in Informaion Retrieval}, SIGIR '03,
  pages 235--242, New York, NY, USA. ACM.

\bibitem[\protect\citename{Ross \bgroup et al.\egroup }2010]{dagger}
St{\'{e}}phane Ross, Geoffrey~J. Gordon, and J.~Andrew Bagnell.
\newblock 2010.
\newblock No-regret reductions for imitation learning and structured
  prediction.
\newblock {\em CoRR}, abs/1011.0686.

\bibitem[\protect\citename{Simon \bgroup et al.\egroup }1997]{simon}
Anik\'{o} Simon, Jean-Christophe Pret, and A.~Peter Johnson.
\newblock 1997.
\newblock A fast algorithm for bottom-up document layout analysis.
\newblock {\em IEEE Trans. Pattern Anal. Mach. Intell.}, 19(3):273--277, March.

\bibitem[\protect\citename{Suzuki \bgroup et al.\egroup }2003]{suzuki}
Masakazu Suzuki, Fumikazu Tamari, Ryoji Fukuda, Seiichi Uchida, and Toshihiro
  Kanahori.
\newblock 2003.
\newblock Infty- an integrated ocr system for mathematical documents.
\newblock In {\em Proceedings of ACM Symposium on Document Engineering 2003},
  pages 95--104. ACM Press.

\bibitem[\protect\citename{Tuarob \bgroup et al.\egroup }2013]{tuarob}
Suppawong Tuarob, Sumit Bhatia, Prasenjit Mitra, and C.~Lee Giles.
\newblock 2013.
\newblock Automatic detection of pseudocodes in scholarly documents using
  machine learning.
\newblock In {\em Proceedings of the 2013 12th International Conference on
  Document Analysis and Recognition}, ICDAR '13, pages 738--742, Washington,
  DC, USA. IEEE Computer Society.

\bibitem[\protect\citename{Zanibbi \bgroup et al.\egroup }2004]{survey}
Richard Zanibbi, Dorothea Blostein, and R.~Cordy.
\newblock 2004.
\newblock A survey of table recognition: Models, observations, transformations,
  and inferences.
\newblock {\em Int. J. Doc. Anal. Recognit.}, 7(1):1--16, March.

\end{thebibliography}
