Alternate {\ttlit ACM
\alignauthor Xi Zheng
\affaddr{The Center for Advanced Research in Software Engineering
% \alignauthor Xi Zheng
% \affaddr{The Center for Advanced Research in Software Engineering
% You can go ahead and credit any number of authors here
% e.g. one 'row of three' or two rows (consisting of one row of three % and a second row of one
two or three). % % The command \alignauthor (no curly braces needed) should % precede each author name
affiliation/snail-mail address and % e-mail address. Additionally
tag each line of % affiliation/address with \affaddr
and tag the % e-mail address with \email. % % 1st. author \alignauthor Ben Trovato\titlenote{Dr.~Trovato insisted his name be first.
This paper provides a sample of a \LaTeX\ document which conforms, somewhat loosely, to the formatting guidelines for ACM SIG Proceedings. It is an {\em alternate style which produces a {\em tighter-looking paper and was designed in response to concerns expressed, by authors, over page-budgets. It complements the document Author's (Alternate) Guide to Preparing ACM SIG Proceedings Using \LaTeX$2_\epsilon$\ and Bib\TeX. This source file has been written with the intention of being compiled under \LaTeX$2_\epsilon$\ and BibTeX. The developers have tried to include every imaginable sort of ``bells and whistles", such as a subtitle, footnotes on title, subtitle and authors, as well as in the text, and every optional component (e.g. Acknowledgments, Additional Authors, Appendices), not to mention examples of equations, theorems, tables and figures. To make best use of this sample document, run it through \LaTeX\ and BibTeX, and compare this source code with the printed output produced by the dvi file. A compiled PDF version is available on the web page to help you with the `look and feel'.
Abstract.tex
We present the {\em Bullseye} system for scholarly search. Given a collection of research papers, Bullseye: 1) identifies relevant passages using any off-the-shelf algorithm; 2) automatically detects document structure and restricts retrieved passages to user-specified sections; and 3) highlights those passages for each PDF document retrieved. We evaluate Bullseye with regard to three aspects: {\em system effectiveness}, {\em user effectiveness}, and {\em user effort}. In a system-blind evaluation, users were asked to compare passage retrieval using Bullseye vs.\ a baseline which ignores document structure, in regard to four types of graded assessments. Results show modest improvement in system effectiveness while both user effectiveness and user effort show substantial improvement. Users also report very strong demand for passage highlighting in scholarly search across both systems considered.
%Researchers nowadays heavily rely on search engines to find relevant works, however, majority of documents returned are irrelevant to the users' search intent. 
%For instance, currently none of the platforms allow a Software Engineer to search for how one specific simulation tool is used in evaluation.
%%{\em Evaluation}
%%{\em implementations} 
%%of one specific simulation tool. Search results from Google Scholar will return the user with research papers which deal with various aspects of the tool and not necessarily how it is evaluated in experiments. 
%This degrades the search experience of the user. 
%%For instance, a Software Engineer researcher might want to find out how one specific simulation tool is used in experiments, when
%%he types the keyword (e.g. the name of the simulation tool) into the Google Scholar, most of the research papers returned are related to the theoretical foundation of the simulation tool
%%or some new ideas on how to improve the simulation tool. The search experience is very frustrating. 
%To help researchers find related information more efficiently and effectively, we look into passage retrieval and leverage pre-defined document structure.
%% to achieve this (e.g. user wants to search the keyword in sections related to {\em Evaluation} only). 
%When passages highly relevant to the search intent are located, our tool highlights those passages within the document in order to preserve the context, instead of just returning the passages themselves. 
%This paper presents the Bullseye algorithm and its implementation, and provides an empirical study using Bullseye to highlight highly relevant passages in the research papers. given a query and scope of the query by the users.

BullsEye.tex
%% LyX 2.0.4 created this file.  For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass{sig-alternate}
\usepackage[T1]{fontenc}
\usepackage[latin9]{inputenc}
\usepackage{color}
\usepackage{verbatim}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{wrapfig}
\usepackage{listings}
\usepackage{mdwlist}
\usepackage{url}
\usepackage{multicol}
\usepackage{enumerate}
\usepackage{caption}
\usepackage{ifpdf}

\ifpdf
   \usepackage{etoolbox}
   \patchcmd{\maketitle}{\@copyrightspace}{}{}{}
   \usepackage{subcaption}
\else
   \usepackage{subfigure}
\fi

\begin{document}
\conferenceinfo{Gold Coast}{'14 Gold Coast, Australia}

\lstset{ %
language=Java,                % choose the language of the code
basicstyle=\scriptsize,       % the size of the fonts that are used for the code
numbers=left,                   % where to put the line-numbers
numberstyle=\scriptsize,      % the size of the fonts that are used for the line-numbers
stepnumber=1,                   % the step between two line-numbers. If it is 1 each line will be numbered
numbersep=5pt,                  % how far the line-numbers are from the code
backgroundcolor=\color{white},  % choose the background color. You must add \usepackage{color}
showspaces=false,               % show spaces adding particular underscores
showstringspaces=false,         % underline spaces within strings
showtabs=false,                 % show tabs within strings adding particular underscores
frame=single,           % adds a frame around the code
tabsize=2,          % sets default tabsize to 2 spaces
captionpos=b,           % sets the caption-position to bottom
breaklines=true,        % sets automatic line breaking
breakatwhitespace=false,    % sets if automatic breaks should only happen at whitespace
escapeinside={\%*}{*)}          % if you want to add a comment within your code
}

%%give 2 line space for initial editing
\linespread{2}

%\title{Bullseye: A Passage Retrieval and Highlight Algorithm Based On Document Structure}
\title{Bullseye: Passage Retrieval and Highlighting for Scholarly Search}
\title{Bullseye: Structured Passage Retrieval and Document Highlighting for Scholarly Search}
%\title{Passage Retrieval and Highlighting for Scholarly Search}
%\title{Bullseye: Highlighting Passages for Scholarly Search}
%\title{Bullseye: Structured Passage Retrieval and Highlighting}
%\title{Bullseye: Passages and Highlighting for Scholarly Search}
%\title{Bullseye: Detecting Document Structure and Highlighting Relevant Passages for Scholarly Search}
%\title{Bullseye: Improving Scholarly Search by Modeling Document Structure and Highlighting Relevant Passages}

 \numberofauthors{2}
 \author{
   \alignauthor Xi Zheng \\
   \affaddr{The Center for Advanced Research in Software Engineering}\\
   \affaddr{The Univeristy of Texas at Austin}\\
   \email{jameszhengxi@utexas.edu}
   \alignauthor Akanksha Bansal, Matt Lease\\
   \affaddr{Department of Computer Science}\\
   \affaddr{The Univeristy of Texas at Austin}\\
   \email{akankshabansal90@gmail.com,ml@utexas.edu}
 }
\maketitle

\begin{abstract}
\input{Abstract.tex}
\end{abstract}


% A category with the (minimum) three required fields
%###\vspace{-5pt}
\category{H.3.4}{Information Systems}{Information Storage and
Retrieval, Systems and Software}
%{Real-time and embedded systems}
%\category{D.2.1}{Software Engineering}
%{Requirements/Specifications}
%\category{D.2.2}{Software Engineering}
%{Design Tools and Techniques - Software Libraries;}
%A category including the fourth, optional field follows...
%\category{D.2.4}{Software/Program Verification}{Assertion checkers, Validation}

%###\vspace{-5pt}
\terms{Algorithms, Experimentation}

%###\vspace{-5pt}
\keywords{Focused Retrieval; Interactive Information Retrieval}

\section{Introduction}
\label{sec:Introduction}
\label{sec:relatedWorks}
\input{Introduction.tex}

%\section{Related Work}
%\input{RelatedWorks.tex}

\input{Implementation.tex}

\section{Experimental Setup}
\label{sec:evaluationSettings}
\input{EvaluationSetting.tex}

\section{Results}
\label{sec:evaluation}
\input{Evaluation.tex}

\section{Limitations}
\label{sec:validity}
\input{Validity.tex}

\section{Conclusion and Future Work}
\label{sec:FutureWork}
\input{Futurework.tex}

%\section{Acknowledgments}
%\input{Acknowledgments.tex}

\bibliographystyle{plain}
\bibliography{library}

\end{document}

EvaluateDocumentRetrievalPassageRetrieval.tex
%% LyX 2.0.4 created this file.  For more info, see http://www.lyx.org/.
%% Do not edit unless you really know what you are doing.
\documentclass{sig-alternate}
\usepackage[T1]{fontenc}
\usepackage[latin9]{inputenc}
\usepackage{color}
\usepackage{verbatim}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{wrapfig}
\usepackage{listings}
\usepackage{mdwlist}
\usepackage{url}
\usepackage{multicol}
\usepackage{etoolbox}
\usepackage{enumerate}

\patchcmd{\maketitle}{\@copyrightspace}{}{}{}


\begin{document}
\conferenceinfo{Gold Coast}{'14 Gold Coast, Australia}

\lstset{ %
language=Java,                % choose the language of the code
basicstyle=\scriptsize,       % the size of the fonts that are used for the code
numbers=left,                   % where to put the line-numbers
numberstyle=\scriptsize,      % the size of the fonts that are used for the line-numbers
stepnumber=1,                   % the step between two line-numbers. If it is 1 each line will be numbered
numbersep=5pt,                  % how far the line-numbers are from the code
backgroundcolor=\color{white},  % choose the background color. You must add \usepackage{color}
showspaces=false,               % show spaces adding particular underscores
showstringspaces=false,         % underline spaces within strings
showtabs=false,                 % show tabs within strings adding particular underscores
frame=single,           % adds a frame around the code
tabsize=2,          % sets default tabsize to 2 spaces
captionpos=b,           % sets the caption-position to bottom
breaklines=true,        % sets automatic line breaking
breakatwhitespace=false,    % sets if automatic breaks should only happen at whitespace
escapeinside={\%*}{*)}          % if you want to add a comment within your code
}

%%give 2 line space for initial editing
\linespread{2}

\title{Bullseye: A Passage Retrieval and Highlight Algorithm Based On Document Structure}

% \numberofauthors{2}
% \author{
%   \alignauthor Xi Zheng \\
%   \affaddr{The Center for Advanced Research in Software Engineering}\\
%   \affaddr{The Univeristy of Texas at Austin}\\
%   \email{jameszhengxi1979@gmail.com}
%   \alignauthor Akanksha Bansal, Matt Lease\\
%   \affaddr{Department of Computer Science}\\
%   \affaddr{The Univeristy of Texas at Austin}\\
%   \email{akankshabansal90@gmail.com,ml@utexas.edu}
% }
\maketitle

\begin{abstract}
\input{Abstract.tex}
\end{abstract}

% A category with the (minimum) three required fields
\category{H.0}{Information Systems}{General}
%{Real-time and embedded systems}
%\category{D.2.1}{Software Engineering}
%{Requirements/Specifications}
%\category{D.2.2}{Software Engineering}
%{Design Tools and Techniques - Software Libraries;}
%A category including the fourth, optional field follows...
%\category{D.2.4}{Software/Program Verification}{Assertion checkers, Validation}

\terms{Algorithms, Experimentation}

\keywords{Passage Retrieval; Empirical Study; Passage Highlight; Document Structure}


\section{Introduction}
\label{sec:Introduction}
\input{Introduction.tex}

\section{Related Work}
\label{sec:relatedWorks}
\input{RelatedWorks.tex}

\section{Implementation}
\label{sec:implementation}
\input{Implementation.tex}

\section{Experimental Settings}
\label{sec:evaluationSettings}
\input{EvaluationSetting.tex}

\section{Experimental Results}
\label{sec:evaluation}
\input{Evaluation.tex}

\section{Conclusion and Future Work}
\label{sec:FutureWork}
\input{Futurework.tex}

%\section{Acknowledgments}
%\input{Acknowledgments.tex}


\bibliographystyle{plain}
\bibliography{library}

\end{document}

EvaluationSetting.tex
%Using our implementation, we have conducted a user study to examine the effectiveness and usefulness of Bullseye.
%\begin{itemize}
%\item Whether Bullseye can improve user satisfaction and effectiveness?
%\item Whether Bullseye can improve the relevance level for the passages retrieved in terms of $quant_{gen}$?
%\item Whether highlighting the retrieved passages, by itself, can improve user satisfaction?
%to make the research contributions consistent
%\item The first will be to see if the implemented system could process arbitrary queries that specify the retrieval element from documents various levels. The purpose will be to validate the model in a qualitative sense and see if the retrieval results are reasonable. 
%\item The second will be to show that our use of the structural information embedded in research documents can actually, improve the effectiveness of document retrieval, in comparison with the case where no such information is used 
%\item Finally, we were interested in finding out whether the idea of giving a bias (in this case highlighting) to an element type actually improves user satisfaction.
%\end{itemize}

%\subsection{Evaluation Metrics}
Following Maskari et al.~\cite{al2010review}, we measured three aspects contribution toward user satisfaction: system effectiveness, user effectiveness, and user effort. In a blind evaluation, users were asked to use and evaluate each system, our Bullseye system and the Baseline system. We also asked users how beneficial passage-level highlighting is for scholarly search.

{\bf System effectiveness} was measured by adopting a focused retrieval metric $quant_{gen}$ from INEX~\cite{lalmas2007evaluating}, which combines two other measurements: \emph{specificity} and \emph{exhaustivity}. Specificity measures how focused the retrieved information is on relevant material and free from other extraneous material. For each highlighted passage retrieved by the system, participants evaluated it on a 4-point graded scale:
\begin{enumerate} [1:]
\setcounter{enumi}{-1}
\item Not specific: the topic of request is not a theme discussed in the element.
\item Marginally specific: the topic of request is a minor theme discussed in the element.
\item Fairly specific: the topic of request is a major theme discussed in the element.
\item Highly specific: the topic of request is the only theme discussed in the element.
\end {enumerate}
% ranging from {\em not specific} (0) to {\em marginally} (1), {\em fairly} (2), or {\em highly} specific (3). 
Exhaustivity, on the other hand, measures how fully the retrieved material covers all aspects of the underlying information need.  Participants judged it on a 3-point scale:
\begin{enumerate} [1:]
\setcounter{enumi}{-1}
\item Not exhaustive: the element did not discuss the query.
\item Partly exhaustive: the element discussed only few aspects of the query.
\item Highly exhaustive: the element discussed most or all aspects of the query.
\end{enumerate} 
%, being {\em not} (0), {\em partially} (1), or {\em highly exhaustive} (2). 
$Quant_{gen}$ slightly favors exhaustivity, assigning high scores to exhaustive, but not necessarily specific elements: 
\[
 quant_{gen} =
  \begin{cases}
   1 	    &  if (e, s) = (3, 3) \\
   0.75   &  if (e, s) \in \{(2, 3), (3, \{2, 1\})\};\\
   0.5     &  if (e, s) \in \{(1, 3), (2, \{2, 1\})\};\\
   0.25   &  if (e, s) \in \{(1, 2), (1, 1)\};\\
   0 	    &  if (e, s) = (0, 0) \\
  \end{cases}
\]

{\bf User effectiveness} was measured by asking participants to report overall satisfaction with the given system being used for ascertaining the relevance of a given document. Users were asked to evaluate satisfaction on a 3-point scale: {\em not satisfied} (1), {\em slightly satisfied} (2), or {\em highly satisfied} (3). 

{\bf User effort} quantifies effort needed to find relevant results (e.g., number of clicks, the number of queries and the number of query reformulations, or rank position accessed to obtain relevant information. We asked participants to quantify their effort in terms of search time required to find the relevant information sought. Participants were asked to report their effort on a 4-point graded scale: {\em highly reduced} (1), {\em slightly reduced} (2), {\em no change} (3), or {\em increased} (4).  

{\bf Highlighting preference.} We also asked users whether integrating such highlighting functionality with Google Scholar search results would improve their satisfaction?  Users were asked this same question regardless of system used in the blind evaluation, using a 4-point scale: {\em worse} (1), {\em no difference} (2), or {\em slightly better} (3), or (4) {\em much better.} 

%We also asked participants to report on a graded scale relative satisfaction with passage highlighting from the given system being used, vs.\ standard document retrieval.  


%For all elements within the highlighted passages, the participants were asked to assess {\emph{Specificity}} based on the below scale:
%\begin{enumerate} [(1)]
%\item Not specific (0): the topic of request is not a theme discussed in the element.
%\item Marginally specific (1): the topic of request is a minor theme discussed in the element.
%\item Fairly specific (2): the topic of request is a major theme discussed in the element.
%\item Highly specific (3): the topic of request is the only theme discussed in the element.
%\end {enumerate}
%For all elements within highlighted passages, the participants were asked to assess their {\emph{Exhaustivity}} based on the below scale:
%\begin{enumerate} [(1)]
%\item Highly exhaustive (2): the element discussed most or all aspects of the query.
%\item Partly exhaustive (1): the element discussed only few aspects of the query.
%\item Not exhaustive (0): the element did not discuss the query.
%\end{enumerate} 

% \begin{enumerate}
% \item {\emph{Specificity}}, which measures the extent to which an element focuses on the topic of the request.
% \item {\emph{Exhaustivity}}, which measures how exhaustively an element discusses the topic of the user's request.
% \end{enumerate}
%For all elements within the highlighted passages, the participants were asked to assess {\emph{Specificity}} based on a graded scale (3-Highly specific,2-Fairly specific,1-Marginally specific,0-Not specific)
% Not specific (0): the topic of request is not a theme discussed in the element.
% \item Marginally specific (1): the topic of request is a minor theme discussed in the element.
% \item Fairly specific (2): the topic of request is a major theme discussed in the element.
% \item Highly specific (3): the topic of request is the only theme discussed in the element.
%For all elements within highlighted passages, the participants were asked to assess their {\emph{Exhaustivity}} based on a graded scale (2-Highly exhaustive,
%1-Partly exhaustive, 0-Not exhaustive).
% 
% \item Highly exhaustive (2): the element discussed most or all aspects of the query.
% \item Partly exhaustive (1): the element discussed only few aspects of the query.
% \item Not exhaustive (0): the element did not discuss the query.
%Our evaluation metric is based on the Exhaustive and Specificity metrics used in INEX 2003 and INEX 2005~\cite{lalmas2007evaluating}. We used the value $quant_{gen}$ to measure the relevant level for the passages that have been deemed to be relevant to the users. 
%% The generalized ($quant_{gen}$)~\cite{lalmas2007evaluating} rewards retrieved elements according to their degree of relevance, thus allowing to reward fairly and marginally relevant elements. 
%$Quant_{gen}$ shows slight preference towards the exhaustivity dimension, assigning high scores to exhaustive, but not necessarily specific elements. 
%It is measured from the values of exhaustivity and specificity calculated based on the above mentioned scales. 

% The relevance metric is based on the Exhaustive and Specificity metrics used in INEX 2003 and INEX 2005~\cite{lalmas2007evaluating}. We used the value $quant_{gen}$ to measure the relevant level for the passages that have been deemed to be relevant to the users. 
%The generalized ($quant_{gen}$)~\cite{lalmas2007evaluating} rewards retrieved elements according to their degree of relevance, thus allowing to reward fairly and marginally relevant elements. $Quant_{gen}$ shows slight preference towards the exhaustivity dimension, assigning high scores to exhaustive, but not necessarily specific elements. It is measured from the values of exhaustivity and specificity calculated based on the above mentioned scales. 

\subsection{User Study}
%Various IR evaluation studies indicate that users are overly impressed with new electronic retrieval technologies, and this accounts for inflated levels of satisfaction with actual search results. 
%In order to account for this bias,  we conducted our tests with users who had wide exposure to various evaluation techniques and were thus be able to provide us with a relatively less biased evaluation of our tool. 
%In the user study, we asked the 

The participant pool was comprised of 15 university students from Software Engineering, Computer Science and Information Science disciplines. Each provided a personal test collection (document sets, content and structural queries, and manually judged set of gold passages). We restricted documents submitted by participants to be within 6-12 pages. Because users familiar with search topics might judge system effectiveness more prudently than unfamiliar users, this could potentially introduce a confounding variable into the evaluation. To mitigate this risk, each also evaluated 5 other query and document sets provided by a different user. 

%Non structured and multi-lingual documents are not in the scope of our user study. 
%We did not consider documents from multiple languages, nor were non-structured documents going made a part of our study. The main reason for our choice was because research documents are fairly long and more importantly can be broken into understandable sections. 
%Our algorithm has been specifically designed for scientific research papers and users of Google Scholar, and we feel that assuming all the research papers retrieved by Google scholar are structured, is safe. 
%Since there is no existing test collection available, which consists of a diverse enough pdf documents, with a set of queries with structural constraints and a gold set of relevant information annotated, we generated the test collection ourselves for the preliminary evaluation and the test collection used for the user study was based on participants input.
%A diverse test collection was generated based on inputs from user study participants.
%Our preliminary evaluation, included processing of arbitrary queries that specified not only a retrieval element at any level in research documents but also some structural conditions so as to validate the algorithm in a qualitative sense. 
%(i.e., to search inside \emph{Implementation} and \emph{Evaluation}) in themselves. 
%The purpose was to validate the algorithm in a qualitative sense and see if the retrieval results were reasonable.
%We downloaded another 50 papers from the same sources as in the training set (e.g., conference papers from Software Engineering), to generate queries containing sections from the documents and annotated the documents
%to create the gold set as relevance judgement.
%The goal in the user study was to answer the following research questions. 
%\begin{itemize}
%\item Whether Bullseye can improve user satisfaction and effectiveness?
%\item Whether Bullseye can improve the relevance level for the passages retrieved in terms of $quant_{gen}$?
%\item Whether highlighting the retrieved passages, by itself, can improve user satisfaction?
%to make the research contributions consistent
%\item The first will be to see if the implemented system could process arbitrary queries that specify the retrieval element from documents various levels. The purpose will be to validate the model in a qualitative sense and see if the retrieval results are reasonable. 
%\item The second will be to show that our use of the structural information embedded in research documents can actually, improve the effectiveness of document retrieval, in comparison with the case where no such information is used 
%\item Finally, we were interested in finding out whether the idea of giving a bias (in this case highlighting) to an element type actually improves user satisfaction.
%\end{itemize}

%15 participants were recruited for this user study. %All participants were in the range of between the ages of 20 and 35 years old. 
%Each participant was required to submit a query containing the keywords and relevant sections they wanted to search for, a document downloaded from Google Scholar by searching the keywords,
%a ``gold'' true optimal passages that have been annotated, which were the most relevant based on the query and the document downloaded.
%Participants will be required to engage in a search task that will require saving one relevant document each for a set of 20 diverse query topics through Google Scholar. 
%Every users would then be sending us the details of the query, the document searched and which section do they want the query output to be returned. 
%We collected this information from each participant to create a test collection.
%They will also provide us with ``gold'' true optimal passages that have been annotated they think is the most relevant section of the document based on their query. 
%We also need to address the issue of bias, which is likely to occur in a real world information retrieval situation, where the user is likely to view proportionally more relevant than irrelevant documents. 
%More specifically,when browsing to look for interesting documents, there is no bias, and the bias increases as a function of the length of a focused search or browsing session. Both extremes are important for a practical proactive information retrieval system. 
%Singhal et al.~\cite{singhal1996document} observed that the likelihood of a document being judged relevant by a user increases with the document length. 
%Based on this information, we restricted the document submitted by the participants to be in range of 6-12 pages.% to rule out possible bias introduced by using documents of various length, which in turn would affect the quality of the passage retrieval as the document itself might carries bias (documents with more pages might have higher chance of being chosen irrespective of relevance level while very small documents containing a very few words have too little information to be useful to a user~\cite{singhal1996document}).

For each participant's submission (document and query), we ran the baseline algorithm and Bullseye to create two separate documents (Algorithm Documents). For each topic (all together 20 topics), there was a set (Test Set) which contains the original document, query, ``gold'' true passages and these two Algorithm Documents generated by the algorithms. 
Each participant was given six Test Sets, with one set containing his own submission. Users then evaluated each Algorithm Document based on the ``gold'' true passages, the document and the query. 
%The evaluation metrics are user effectiveness, user effort, specificity and exhaustivity as covered previously. 
%stated that length of the documents affects the relevance of the information. 
%Based on this information, in the user study we will ask the participants to use documents which are at least one page in length but no more than 10 pages excluding reference page. 
%However, if the document is very lengthy, participant might stat loosing interest and skimming through the document will become difficult. It will also be an unnecessary burden on their memory as they moved through the document trying to identify and track relevant information. Therefore the document length size will be restricted to maximum of 10 pages. 
%For each participant, we will create 
%Each participant will be given six sets of 
%the document he provided, the query, two other documents where one of them will be generated by the baseline algorithm and the second one will be generated by Bullseye.  The users will not know as to which of the two documents has been generated after running which algorithm. 
%The users will be returned the documents they had provided, with two other documents where one of them will be generated after running the baseline algorithm and the second will be generated after running the Bullseye algorithm.
%They will then be asked to rate their satisfaction with regards to comparison of the both the documents. Users will be returning their results in a 3-point scale: 
%\begin{enumerate} [(i)]
%\item Document A/B is much better than Document A/B 
%\item Document A/B is slightly better than Document A/B
%\item Document A is same as Document B
%\end{enumerate}
%Their evaluation will test our research hypothesis that if the search engine had provided them with the highlighted text document, it would have saved them a lot of time. They will also be asked some questions as the end of this user study so that we can get an idea about what is the expectations of users from the search engines when it comes to structured data retrieval. 

Before the user study, we organized a training session with all participants reviewing aims of our study, evaluation metrics, evaluation process, and participant responsibilities. 
Each participant was made aware that the Test Set to be provided consisted of two Algorithm Documents with relevant passages highlighted by two alternative algorithms. We explained evaluation questions being asked. In this manner, each user was trained how to create gold set by himself and compare the gold set with the algorithms' output.
 
%And If this tool were to be integrated with the Google Scholar search results, would it improve their satisfaction with the search results ?
%The participants were asked to assess this based on the below scale:
%\begin{enumerate} [(1)]
%\item Much better  
%\item Slightly better
%\item No difference
%\item Worse
%\end{enumerate}

Evaluation.tex
%As for any tool designed to enhance user experience, our evaluation was based on measuring user satisfaction. 
%With below evaluation metrics , we tried to capture if users feel that Bullseye enhances their satisfaction with a search engine.% (e.g. Google Scholar). 
%The user study helped us understand how effectively our algorithm improves user's experience by highlighting relevant passage. 
%Using our implementation, we have carried out an empirical evaluation to answer the following questions about the effectiveness and usefulness of Bullseye. 

%\subsection{Empirical Design}


%We asked one more question for the participants to justify our future work: 
%\begin{enumerate} [(a)]
%\item If this tool were to be integrated with the Google Scholar search results, would it improve your satisfaction with the search results ?
%\item If yes, then would such a tool be helpful to you be a great amount, that is will it reduce the time you have to spend in performing various searches ?
%\item Do you think that if this tool was extended to all the search engines, that is this tool is integrated with search being done in Google as well and not specifically, will it be helpful?
%\end {enumerate}

%We want to keep the test setup required by the participants as simple as possible and therefore we expect to follow  is the evaluation procedure we expect to follow. 
%We call a combination of one query and the most relevant document for that topic and the ideal text that needs to be highlighted a  initial Query Document set. Once we have run the two algorithms on the document we generate a complete query Document set ready for evaluation from participants. Each user will be completing 25 such query document sets, with a different search topic and documents in each.

%The users will be asked to email us 20 documents, their corresponding query and the test which they think needs to be highlighted. These details will be capture in an excel sheet for each of the user. Then we will be emailing the participants the documents set for each of their query along with 5 more document and query set along with the excel sheet where they need to record their evaluation results. They will be emailing us their feedback in two days of time. 
%We wanted to keep the test setup required by the participants simple and less time consuming so we asked the participants to email us the submission(document, query and the gold set) instead of having to come in person to a controlled environment for the user study.
%After having generated the Test Sets for all the topics, we then emailed the participants the six Test Sets and survey sheet for the measurement. The participants filled in six of theses surveys, based on which we generated the following graphs to analyze the results of the user study:
%Based of the results which have been provided by the participants we will be generating the following graphs to depict the results of the study : 
%\begin{itemize}
% \item A Box-and-Whisker Plot with the details of User Effort calculated for both Bullseye and baseline document. 
% \item A Box-and-Whisker Plot with the details of User Effectiveness calculated for both Bullseye and baseline document. 
% \item A Box-and-Whisker Plot for the $Quant_{gen}$ calculated for both Bullseye and baseline document.
% \item A Box-and-Whisker Plot with the details of the users feedback for highlighting the passage retrieved within the document. 
%\end{itemize}


%We evaluated how Bullseye enhances the relevance measurements using 
%rposes.  For each of our three applications, we evaluated the benefits of using BraceForce (specifically in terms of reduction in development time and development effort) and the potential negative ramifications of using BraceForce to interface with the sensors (in terms of the accuracy of the sensing task and the running time required to navigate the layered framework).  Fig.~\ref{fig:Q1} shows the results.
%On the right hand, we see the significant {\em development} benefit that BraceForce brings to programming sensor-driven mobile applications.  For the simple Application 1, BraceForce had little measurable benefit to the development time and lines of code.  This application was exceedingly simple to develop using either approach.  The story is completely different for Applications 2 and 3.  Specifically, BraceForce resulted in a 64-99\% reduction in the average total number of lines of code that an application developer had to write and a 67-98\% reduction in the average total time required to write them.  The reductions are because BraceForce provides much of the reusable effort within the framework and BraceForce explicitly separates the concerns associated with programming at the sensor driver level and at the level of the application logic.
%These benefits did not come at zero cost.  While the accuracy of the two versions of all three applications was not noticeably different (it was actually marginally better when the programmer used BraceForce), the chart on the left of Fig.~\ref{fig:Q1} shows that executing through the BraceForce tiered architecture does have a measurable impact on the application's execution efficiency (in terms of how long it took the application to run, including interacting with the sensors).  However, while this burden was high (a 20\% increase) for Application 1, it was substantially lower for the more sophisticated Applications 2 and 3, since the interaction with the BraceForce architecture is amortized over more extensive and potentially distributed interactions.

{\bf System Effectiveness.} Fig.~\ref{fig:Quant} shows the box-and-whisker plot of $quant_{gen}$ score distributions for Bullseye vs.\ the Baseline (no structural filtering). While Bullseye's average improvement is modest (0.47 vs.\ 0.41 mean), its score distribution skews much higher. The lower quartile (Q1) is almost equivalent for the Bullseye and Baseline (0.26 vs.\ 0.25). but in the upper quarter (Q3), Bullseye is much better (0.75 vs.\ 0.5). 
In some cases, though, sentences matching the query term happen to exist only in the user-specified sections, in which case structural filtering has no effect. 

{\bf User Effort.} Fig.~\ref{fig:UserEffort} shows the box-and-whisker plot score distribution of Bullseye vs.\ Baseline. We see that Bullseye substantially reduces user effort vs.\ the Baseline, indicating users reported far less effort involved when using Bullseye in the blind evaluation. Reductions include the mean (1.61 vs.\ 2.22), Q1 (1.0 vs.\ 2.0), and Q3 (2.0 vs.\ 3.0). 

{\bf User Effectiveness.} Fig.~\ref{fig:UserEffectiveness} shows the box-and-whisker plot score distribution for Bullseye vs.\ the Baseline. We see that users report being much more satisfied with Bullseye in the blind evaluation in the mean (2.33 vs.\ 1.76), Q1 (2.0 vs.\ 1.0), and in Q3 (3.0 vs.\ 2.0). 

{\bf Highlighting preference.} Participants were unanimous across both systems in indicating that integrating such highlighting would yield much better satisfaction. While we do not learn anything about relative performance of one system vs.\ the other from this (and so do not show a corresponding boxplot), we observe clear and very strong demand for supporting passage-highlighting in scholarly search. % overall, regardless of system used.  


% \begin{figure}[htb]
% \center{\includegraphics[width=8cm, height=4cm]{Quant.png}}
% \vspace{-.75cm}
% \caption{\label{fig:Quant} $quant_{gen}$ metric}
% \vspace{-0.75cm}
% \end{figure}
% 
% \begin{figure}[htb]
% \center{\includegraphics[width=8cm, height=4cm]{UserEffectiveness.png}}
% \vspace{-.75cm}
% \caption{\label{fig:UserEffectiveness} User Effectiveness metric}
% %\vspace{-0.75cm}
% \end{figure}
% 
% \begin{figure}[htb]
% \center{\includegraphics[width=8cm, height=4cm]{UserSatisfaction.png}}
% \vspace{-.75cm}
% \caption{\label{fig:UserSatisfaction} User Satisfaction metric}
% \vspace{-0.75cm}
% \end{figure}

\begin{figure}[htb]
\captionsetup[subfigure]{aboveskip=-1pt,belowskip=-1pt}
\centering
%\ifpdf
\begin{subfigure}[b]{2.7cm}{
 \includegraphics[width=2.7cm]{Quant.png}
 \caption{$quant_{gen}$}
 \label{fig:Quant}}
\end{subfigure}
\begin{subfigure}[b]{2.7cm}{
  \includegraphics[width=2.7cm]{UserEffort.png}
   \caption{Effort}
  \label{fig:UserEffort}}
\end{subfigure}
\begin{subfigure}[b]{2.7cm}{
  \includegraphics[width=2.7cm]{UserEffectiveness.png}
  \caption{Effectiveness}
  \label{fig:UserEffectiveness}}
\end{subfigure}
%\fi
\caption{Evaluation Metrics}
\label{fig:figure}
\vspace{-0.7cm}
\end{figure}


% In the user survey, we also asked the participants to give us feedback whether highlighting the passage retrieved in the document is better than simply returning the passage retrieved.
% Participants gave us an unanimous answer, highlighting is ``Much Better''.  We conclude that by maintaining the context of passage retrieval, users can have better understanding of the information
% returned, therefore is able to satisfy their original search intents.

%\subsection{Validity Discussions}
%We have made some assumptions in the implementation and the empirical design, which we describe below and we also talk about how we manage to mitigate the validity threats associated with them.

%User study has been users to substantiate our algorithm effectiveness in improving user satisfaction by highlighting most relevant passages in the research papers. 
%The following assumptions have been made in the implementation and our user study was able to prove that these assumptions do not hinder the performance of our algorithm. 

%First of all, the additional search condition for sections is a fixed set composed of Abstract, Introduction, Related Work, Implementation, Evaluation, and Future Work/Conclusion. 
%We have an assumption that in general, the structure of most research papers shall be similar to the Fixed Set. %Using the training set and the user study, this assumption is supported. However, 
%The number of research papers this hypothesis was tested upon is limited, papers outside of our test study might exhibit very different section structure, which can invalidate the Bullseye sections location algorithm. 
%However, in these cases, Bullseye at least can perform the same as the Baseline, which is not bad from the evaluation metrics.
%, as we have built the logic into the Bullseye to detect section location matching failure and resort back to Baseline in this exceptional condition.

%Second validity threat is the fact that decision trees used by the Bullseye is based on manual analysis of the training set. 
%Though we have attempted to use automatic decision tree induction process, we found the manual process is much simple to implement and more appropriate for our research scenario than the two decision tree induction techniques~\cite{umanol1994fuzzy}~\cite{ruggieri2002efficient}  tried. 
%It might be possible that automatic tools for decision trees are more suitable to locate section headers, we argue due to the fixed set of the section headers we defined for the search condition, the current decision trees generated
%is sufficient.

%First of all the implicit section location algorithm of Bullseye is based on manual analysis of the training set. The derived rules are implemented in the form of decision trees.
%This could result in some exceptions to the decision trees introduced by some other documents not included in the training set. However..... 
%Another validity threat is the bias in creating the test collection as the test collection might be not representative enough of general research papers.  
%To mitigate this threat, we asked the participants from different domain to prepare their own topics and create gold true passages themselves.

%Finally to avoid users giving us a biased opinion to the Bullseye algorithm, we alternated the document name for the Baseline and the Bullseye so as to ensure that the participants
%had no idea of the correlation between documents and algorithms.  We gave each user six test suites, including one from the user himself, to avoid learning effect.


Futurework.tex
We are motivated by the fact that a user satisfaction can be enhanced by improving the overall search experience. We delivered this by implementing Bullseye algorithm which is able to 
narrow down the search within user-specified sections and highlight highly relevant passages within the document. The results show Bullseye is able to improve
user satisfaction and search effectiveness. 

We wish to port Bullseye algorithm to a Google Chrome Application so after a user retrieves the documents from a search engine (e.g. Google Scholar), the browser automatically highlight the most relevant information to the user. %This shall provide a seamless integration of information retrieval process.
%This would allow the user to be able to search very effectively. 
%As part of the feedback of our user study, many of the participants commented that they have felt the need of such an interface. 
%Currently whenever they find any relevant document via the search engine,  the first thing that they have to do is to search in the documents for the exact words they had typed in the query. 
%This tool would be able to intelligently help the decision for the user about the relevance judgement of the given document. 
%The application would have a feature where the user can specify some additional search conditions (e.g., which sections of the paper the user is more interested in). 
%The application will provide a seamless integration of document retrieval (e.g., provided by the search engine) and passage retrieval (e.g., provided by the Bullseye) in the Chrome Browser. 
%, we believe it shall be a better user experience. 
%However we will need to enhance the application to be able to highlight text for different kinds of documents. 
%The application would be able to highlight not only the text which is present in structured documents (e.g., research papers) but also other forms of documents. 

%We also wish to explore if there exist other options where a browser might be able to enhance the search experience. 

%Another example of this kind of enhancement could be the browser allowing the user to goto the next search result if he is unsatisfied with results on the current page. 

\section*{Acknowledgments}

This study was supported in part by National Science Foundation grant No. 1253413, DARPA Award N66001-12-1-4256, and IMLS grant RE-04-13-0042-13. Any opinions, findings, and conclusions or recommendations expressed by the authors are entirely their own and do not represent those of the sponsoring agencies
Implementation.tex
%For the baseline algorithm, we presume the user has already got a list of relevant documents for the query from a popular search engine like Google Scholar. 
%So the main implementation lies in applying a search algorithm which highlights the relevant parts in the documents.

%Based on Tellex's \emph{evaluation of passage retrieval algorithms for Question Answering}~\cite{tellex2003quantitative}, we implemented a passage retrieval algorithm based on the MITRE~\cite{light2001analyses}.
%This approach simply matches stemmed words between question and answers. 
%It counts the number of terms a passage has in common with the query, where each sentence is treated as a separate passage. 
%This algorithm represents the simplest reasonable passage retrieval technique and serves as a good baseline for comparison. 
%%Our implementation of the baseline algorithm is based of MITRE. 
%In the baseline algorithm implementation, we also remove the stop words and aggregate sentences where query terms span across.

\begin{figure}[!hbt]
%\vspace{-1cm}
\begin{center}
\ifpdf
\includegraphics[width=1.1\columnwidth, trim=8cm 0cm 0cm 5cm]{DecisionTree.png}
\else
\includegraphics[width=1.1\columnwidth]{DecisionTree.png}
\fi
\end{center}
\vspace{-1.25cm}
\caption{Decision Tree}\label{fig:TE}
\vspace{-0.5cm}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{The Bullseye System}

%Bullseye is designed to be able to locate section headers as specified by the users to narrow down the search (This is based on the assumption when searching for keywords in the research papers, users have a clear idea of what sections to look for).

Given a collection of PDF documents, Bullseye first pre-processes documents to extract text, detect document sections, and map observed section headers to a fixed set of target sections. To search, Bullseye takes two key inputs: a search query and a listing of any document sections to which results should be restricted. Bullseye does not define a new passage retrieval algorithm but can incorporate any such method off-the-shelf. In this work, we adopt a very simple algorithm developed by MITRE~\cite{light2001analyses}, which treats each sentence as a passage. Following sentence boundary detection, stopwords are removed~\cite{frank2000text}, and terms are then stemmed~\cite{croft1994corpus} and matched between each query and its passage candidates. We then filter the set of retrieved passages so that any found outside the user's listing of specified sections are discarded. 

Finally, relevant passages in the PDF documents are highlighted using Apache PDFBox~\cite{PDFBox}. While we had expected this to be straightforward, finding exact locations of the text in the PDF turned out to be non-trivial. To accomplish this, we: 1) mark the beginning of each article, page, paragraph, and word; 2) determine the default text orientation in the document (e.g., the glyph is written from left to right or from right to left); determine the default document structure (e.g., single column vs multi column); determine the default font size and spacing (e.g., line spacing and word spacing); and finally 4) detect overlapping text.

\subsection{Detecting Document Structure}

Our approach is inspired by Denny et al.~\cite{denny2009evaluation}, whose {\em SecTag} algorithm identifies both labeled and un-labeled (implied) section document headers using Bayesian statistics. We first created a training set of 200 research papers from a variety of software engineering conferences: FSE, ICSE, ASE, ICCPS, MobiCom, and OOPSLA. Manual inspection showed that section headers can be fairly broadly defined by a fixed set of {\em Abstract, Introduction, Related Work, Implementation, Evaluation, Conclusion} and {\em Future Work} sections. %. \{referenced as Fixed Set\}
%the following sets to satisfy users' information need. 
%\begin{itemize}
% \item {\em Abstract}:  It usually contains high level information regarding the keywords searched by users.
% \item {\em Introduction}: It often provides context and problem domain of the keywords searched by users.
% \item {\em Related Work}: It is highly likely to locate the state of the art for the keywords searched by users.
% \item {\em Implementation}: It usually provides how the keywords searched are used in the implementation.
% \item {\em Evaluation}: It is likely to provide information of how the keywords are applied in the experimentation.
% \item {\em Conclusion/Future Work}: It might give novel ideas of how the keywords can evolve in the future.
%\end{itemize}
%We defined this set (Fixed Set) which the user provides in the search condition for section headers.
Many papers we inspected had section headers which were either exact matches or equivalent terms (e.g. \emph{Summary} is equivalent to \emph{Conclusion/Future Work}). Some papers did have unusual section headers which do not match explicitly with headers in the target set and could not be clearly mapped. 
However, since users would likely not know to query for these sections either, this seems to be a relatively minor concern. 
The following observed patterns were incorporated into Bullseye (as deterministic decision trees) for matching observed section 
headers to target headers.

\begin{enumerate} [(1)]
 \item Section headers largely respect implicit ordering. %(e.g. \emph{Implementation} is always before \emph{Evaluation}).
 \item Observed section headers often exhibit a many-to-one relationship to our target set of sections. %(e.g. two implicit sections is mapped to \emph{Implementation}).
 \item Observed section headers tend to fall into one of three target section headers : \emph{Related Work}, \emph{Implementation} and \emph{Evaluation}.
 \item Not all target section headers are available in all papers. %(e.g. Some research papers simply do not have \emph{Evaluation}).
% \item \emph{Related Work} section either appear before the \emph{Implementation} or after \emph{Evaluation}. But in all explicitly matched scenarios, \emph{Related Work} always appears before \emph{Implementation}.
 \item \emph{Implementation} and \emph{Evaluation} target section often match multiple observed sections
 \item \emph{Related Work} is less likely to have multiple observed section headers matched to it, compared with \emph{Implementation} and \emph{Evaluation}.
\end{enumerate}

Bullseye first processes the document, sentence by sentence, to find all possible section headers. A section header is one that marks the beginning of sentences and consists of only Capital letters and a leading number or Roman numbers. %(from manual observation, we found all the papers in the training set have major sections defined in this format).
All the possible section headers are inserted into a linked list which preserves the insertion order. Each section header candidate is matched against the equivalent terms for sections defined in the target set. The matched results are recorded into the linked list. Bullseye then checks in the linked list whether it can find matches for all the section search conditions specified by the user. If it could not locate all matching sections, Bullseye uses the decision trees based on the ordering and statistical information found previously to locate section headers specified by the user. 

Fig.~\ref{fig:TE} shows {\em one} of the learned decision trees in the Bullseye algorithm. This decision tree specifically is used to determine implicit \emph{Related Work} and \emph{Implementation} when \emph{Evaluation} section is found through explicit matching process.  The decision process
conforms to training set statistics. Through the explicit and implicit section header matching, Bullseye maintains a state machine of how many potential section headers have been explicitly matched and how many are implicitly matched. The intersection of matched section headers in the state machine and user specified section query condition are used to pinpoint those sections for locating the keywords submitted in the user query.
%\subsection{Tools Used}
%KSTEM~\cite{croft1994corpus}  and Frank et al~\cite{frank2000text} were used in our implementation. 
%KSTEM~\cite{croft1994corpus} is an open source Java tool to stem words and Frank et al~\cite{frank2000text} developed open-source software which removes stop words. 
%We used both in our implementation.
%The Apache PDFBox~\cite{PDFBox} library was used to extract sentences from PDF documents and highlight sentences in the documents.
% \subsection{Idiosyncrasies of PDF Highlighting}
% %Highlighting is a form of secondary notation, since the highlights are not part of the text meaning, but serves to reinforce it. %It is one of our research hypothesis that it improves the readability and context of the text. 
% %However, we encountered a few unexpected challenges implementing this feature using the selected library, which we describe below.
% %The Apache PDFBox~\cite{PDFBox} library allows us to perform various operations on the pdf documents. It allows for quick and effective manipulation of existing pdf documents with the ability to extract content from documents. 
% %However, 
% We used the Apache PDFBox library~\cite{PDFBox} to extract sentences from PDF documents and highlight sentences in the documents.
% However, we found out to highlight data from a pdf, 
% %In order to be able to highlight data from a pdf, %we needed to write code base to extract the various elements of the pdf document and then highlight the sections which we deemed to be relevant for the user's query. 
% %In order to do so, our pdf Highlighting API 
% we need non-trivial computation to extract more information (e.g., the beginning of the text, the default document orientation, the font size and spacing, and text overlapping) to pinpoint the locations of the text which our BullsEye algorithm deems to be the most relevant derived from the user's query. 
%it is essential to recognize the locations of the text which our BullsEye algorithm deems to be the most relevant derived from the user's query. 
%Getting information of the exact locations of the text involves non-trivial computations. 
%We listed a few computation required and these unexpected engineering efforts might help future research requiring highlighting inside a PDF.
%The following computation were required to determine the exact locations of the text:
%\begin{itemize}
%\item Determine the beginning of a page, of an article, of a paragraph, of a word. 
%\item Determine the default text orientation in the document (e.g., the glyph is written from left to right or from right to left). 
%\item Determine the document structure (e.g., single column vs multi column).
%\item Determine the font size and spacing (e.g., line spacing and word spacing). 
%\item Recognize text overlapping %and still be able to highlight the text.  
%\end {itemize}
%We added these features on top of the Apache PDFBox~\cite{PDFBox} library, we added these features to 
%These features were built on top of the Apache PDFBox library~\cite{PDFBox}.
%to improve the efficiency of the algorithm, we used a separate thread to preprocess the document to acquire these information while allocating the main thread for the Bullseye to determine the best passage based on the user's query conditions.  
%while the Bullseye uses the main thread to determine the best passage based on the user's query conditions.  

Note that we have built logic into Bullseye to detect massive section location matching failure and resort back to ignoring in this exceptional condition.

Introduction.tex
While scholarly search engines like Google Scholar and Microsoft Academic Search enable quick and easy access to past research papers, search continues to be performed at the document level. This requires users to search through documents manually to find relevant material, or to determine that a document was not relevant after all. In contrast, {\em passage retrieval}~\cite{salton1993approaches,kaszkiel1997passage} and {\em focused retrieval}~\cite{trotman2007report} methods seek to identify relevant portions of the document to ease this burden. While one could return only passages, Kamps et al.~\cite{kamps2003xml} found out that users and assessors still regard whole articles as the meaningful unit of retrieval. Consequently, we develop and evaluate an approach of highlighting of relevant passages~\cite{knaus1996highlighting,milic2005facility,allan2005will} in scholarly documents.

%Search engines like Google have made the lives of researchers easy by providing them with specialized search system like Google Scholar.
%Now one can find relevant work being done in any of the domains. 
%However, analyzing the relevance of documents takes much more time because a researcher has access to huge number of repositories. 
%Additionally, one needs to go over the whole document just to realize that the document returned is irrelevant, which becomes frustrating with time. 
%In this paper, we present a tool that is able to help users in making the said decision. 

%Passage retrieval based on structural information in documents has long been suggested as an effective way to retrieve elements of a document with finer granularity. 
%With the help of passage retrieval algorithms, we want to highlight the text that is relevant to the user based on the query, thus making it easier for him to understand the document~\cite{knaus1996highlighting,milic2005facility,allan2005will}. 


%We posit that the search engine will not only return the user the best ranked documents but will also be part of the user decision making process. 
%One might argue that if the search engine just returns passages which it deems to be relevant instead of a document, 
%then such a tool is irrelevant. Because then the user would be able to find relevant material much faster. 
%But this approach leads to loss of context with regards to the usage of the data. 
%Kamps et al.~\cite{kamps2003xml} found out that users and assessors still regard whole articles as the meaningful unit of retrieval for XML collection, which adds to our intentions (e.g. to highlight the relevant passages within a document instead of
%just returning them).

In scholarly search, section structures in research papers provide important context for locating relevant information, which either the system may exploit in search, or the user may specify in expressing section-level relevance criteria for search. For example, Lalmas et al.~\cite{lalmas2007evaluating} distinguish {\em Content-only} (CO) INEX topics, in which users do not express structural relevance criteria, vs. {\em Content-and-structure} (CAS) topics, in which users express structural constraints. 
%\begin{itemize}
%\item {\em Content-only (CO) } topics are requests that do not include reference to the document structure. They are traditional topics used in information retrieval test collections. However, the results to such topics are elements of various complexity, e.g. at different levels of the XML documents structure.
%\item {\em Content-and-structure (CAS)} topics are requests that contain conditions referring both to content and structure of a document. These conditions may refer to the content of specific elements, or may specify the type of the requested answer elements (e.g. sections should be retrieved).
%\end{itemize}
%For example, the section \emph{Related work} contains the state of the art literature review in the field which the research paper addresses, while \emph{Evaluation} contains the test methodologies and tools used to support the main research hypotheses and findings in the paper
Our review of a few decades of past research papers suggests that common section headers have only changed minimally over time, suggesting an abundance of training data and generality of techniques. Extracting this implicit structure across varying document formats is an open challenge (cf.~\cite{denny2009evaluation}).



In this work, we investigate the following research questions in the specific domain of scholarly search. 
%How can we classify and localize the structural information embedded in documents for passage-level retrieval?  How can we improve user satisfaction by combining content and structural features in passage-level retrieval?  How can passage highlighting improve user satisfaction?
\begin{itemize}
\item How can we classify and localize implicit structure in scholarly documents for passage retrieval?  
\item How can we improve user satisfaction by combining content and structural features in passage-level retrieval?  
\item How can passage highlights improve user satisfaction?
\end{itemize}

%If the user is interested in the current status quo of the keyword, he most likely shall be able to locate the information in the \emph{Related work}
%while the user shall be more likely to find information in \emph{Evaluation} if he is particularly interested in how the keyword is related to experiments.
%Thus the scope of our research is focused on answering the below research questions:
%\begin{itemize}
%\item How to classify and localize the structural information embedded in documents for passage-level retrieval?
%\item Whether a retrieval system can improve user satisfaction by combining content and structural conditions in passage-level retrieval?
%\item Whether a retrieval system can enhance search effectiveness by combining content and structural conditions in passage-level retrieval?
%\item Whether highlighting passage retrieved can improve user satisfaction?
%%the research questions shall not be vague, shall be very explicit and answered later in implementation and evaluation
%%\item how the structural (sometimes discourse-level) information embedded in documents can be exploited to enhance effectiveness of document-level retrieval. 
%%\item how a retrieval system can provide users with a variety of access points to documents in response to queries that express both the content and the structure requirements 
%%\item can combining document-level and passage-level improve user satisfaction. 
%\end {itemize}

The {\em Bullseye} system developed in this work adopts an off-the-shelf passage retrieval algorithm by MITRE~\cite{light2001analyses}, but uses detected section structure information to restrict results to sections specified by the user. To highlight relevant passages in PDF documents, we utilize the Apache PDFBox~\cite{PDFBox} library. While we had expected this to be straightforward, finding exact locations of the text in the PDF turned out to be non-trivial, as further discussed later.


Following Maskari et al.~\cite{al2010review}, we measure three aspects contribution toward user satisfaction: {\em system effectiveness}, {\em user effectiveness}, and {\em user effort}. In a system-blind evaluation, users were asked to compare Bullseye highlighting vs.\ structure-agnostic MITRE highlighting. System effectiveness was measured by adopting a focused retrieval metric $quant_{gen}$ from INEX~\cite{lalmas2007evaluating}, which combines two other measurements: \emph{specificity} and \emph{exhaustivity}. Users reported each on a graded scale. User effectiveness was measured by asking participants to report overall satisfaction on a graded-scale. Finally, user effort asked users to quantify the amount of time saved (or lost) on a graded scale. While results show only modest improvement in system effectiveness, both user effectiveness and user effort show substantial improvement.



%The first step consisted of building a baseline passage retrieval algorithm to highlight the data which are most relevant to users (documents are research papers) based on their queries. 
%We want to implement the passage retrieval algorithm based on MITRE~\cite{tellex2003quantitative}~\cite{light2001analyses}. 
%The variance that we will have in this algorithm is that the search algorithm will stem words and remove stop words. 
%As a second step, we built the Bullseye algorithm on top of the baseline algorithm. It highlights the relevant text only from the sections of the document that user wants to find. 
%Search engines nowadays are anyways generating structure maps after analyzing the documents so as to be able to decide if the given document is relevant or not. 
%Bullseye algorithm shows how search engines can allow the user to specify which section of the document he wants the results from. 
%As part of Bullseye algorithm, we are proposing that they allow the user to specify which section of the document he wants the results from. 
%Taking the example of a user of Google Scholar, a researcher might be looking for the implementations of a specific technology. Each research %paper has some components to it like: 
%Abstract/Summary, Implementations, Conclusion, References etc. It is possible that Goggle Scholar has returned a specific paper only because %%that paper has mentioned the 
%technology in the query in the References section. A user would first be spending a lot of time figuring out if the paper has any reference %%%to his query and then would be spending 
%his time deciding if the papers implementation section has any reference of what he was looking for.  
%Search engines nowadays are anyways generating maps after analyzing the documents so as to be able to decide if the given document is relevant or not. 

%The next section places this paper's contributions in the context of related work. In Section 3, we provide complete details of the Bullseye algorithm. 
%Section 4 presents the empirical design for our evaluation, Section 5 describes its results, and Section 6 discusses some validity issues.

RelatedWorks.tex
%FROM:Hildreth, C. R. (2001) Accounting for users? inflated assessments of on-line catalogue search performance and usefulness: an experimental study. Information Research.
%Hildreth et al.~\cite{hildreth2001accounting} argued that evaluation studies that relied on measures such as user perception of ease of use and subjective satisfaction with the search results did not provide a clear and consistent answer as to how user satisfaction may predict their actual search effectiveness. He found that user perception of ease of use had an effect, possibly greater than the results themselves, on user satisfaction.
Kamps et al.~\cite{kamps2003xml} listed two ways of Information Retrieval, which are {\em Full Document Retrieval } 
and {\em Element Retrieval }
%can be performed for text : 
%\begin{itemize}
%\item {\em Full Document Retrieval System } A baseline is formed by using a standard document index in which only whole documents are considered as a retrievable unit.
%\item {\em Element Retrieval System } The documents are indexed into retrieval units. Elements rather than documents then become the returned results of the user query. 
%\end{itemize}
In this paper, we concentrate on passage-level retrieval and highlight. In the measurement, we not only cover user perception of ease of use and satisfaction with the search results,
but also cover more objective metrics (e.g., exhaustivity and specificity as covered in Section~\ref{sec:evaluationSettings}).
%not very clear of how the works above are related to ours
%With our research we wish to establish that combination of document-level and passage-level improves user satisfaction.~\cite{model} ~\cite{kamps2003xml} 

Lalmas et al.~\cite{lalmas2007evaluating} mentioned two types of topics being identified by INEX,
where are {\em Content-only (CO) } and  {\em Content-and-structure (CAS)}.
%\begin{itemize}
%\item {\em Content-only (CO) } topics are requests that do not include reference to the document structure. They are traditional topics used in information retrieval test collections. However, the results to such topics are elements of various complexity, e.g. at different levels of the XML documents structure.
%\item {\em Content-and-structure (CAS)} topics are requests that contain conditions referring both to content and structure of a document. These conditions may refer to the content of specific elements, or may specify the type of the requested answer elements (e.g. sections should be retrieved).
%\end{itemize}
The baseline algorithm is closely related to answering queries in INEX Content-only(CO) topics where the user requests do not include reference to the document structure. The Bullseye algorithm is closely related to answering queries in INEX Content-and-structure topics where the user requests do contain conditions 
referring both to content and structure of document.

Cui et al.~\cite{cui2003hierarchical} proposed a new hierarchical index propagation and pruning mechanism for structured documents and realized a flexible element retrieval system based on this index structure. 
Their work was carried on XML documents, whereas our research focuses on PDF documents. 

Denny et al.~\cite{denny2009evaluation} designed a SecTag algorithm to identify both labeled and un-labeled (implied) note section headers in ``history and physical examination'' documents.
Their work calculated Bayesian probability for all of a document's section header candidates while our work is much more straightforward as each section header in a research paper is labeled. 
Our work of automatically identifying section headers is largely built upon the SecTag's method.
sig-alternate.tex
% This is "sig-alternate.tex" V2.0 May 2012
% This file should be compiled with V2.5 of "sig-alternate.cls" May 2012
%
% This example file demonstrates the use of the 'sig-alternate.cls'
% V2.5 LaTeX2e document class file. It is for those submitting
% articles to ACM Conference Proceedings WHO DO NOT WISH TO
% STRICTLY ADHERE TO THE SIGS (PUBS-BOARD-ENDORSED) STYLE.
% The 'sig-alternate.cls' file will produce a similar-looking,
% albeit, 'tighter' paper resulting in, invariably, fewer pages.
%
% ----------------------------------------------------------------------------------------------------------------
% This .tex file (and associated .cls V2.5) produces:
%       1) The Permission Statement
%       2) The Conference (location) Info information
%       3) The Copyright Line with ACM data
%       4) NO page numbers
%
% as against the acm_proc_article-sp.cls file which
% DOES NOT produce 1) thru' 3) above.
%
% Using 'sig-alternate.cls' you have control, however, from within
% the source .tex file, over both the CopyrightYear
% (defaulted to 200X) and the ACM Copyright Data
% (defaulted to X-XXXXX-XX-X/XX/XX).
% e.g.
% \CopyrightYear{2007} will cause 2007 to appear in the copyright line.
% \crdata{0-12345-67-8/90/12} will cause 0-12345-67-8/90/12 to appear in the copyright line.
%
% ---------------------------------------------------------------------------------------------------------------
% This .tex source is an example which *does* use
% the .bib file (from which the .bbl file % is produced).
% REMEMBER HOWEVER: After having produced the .bbl file,
% and prior to final submission, you *NEED* to 'insert'
% your .bbl file into your source .tex file so as to provide
% ONE 'self-contained' source file.
%
% ================= IF YOU HAVE QUESTIONS =======================
% Questions regarding the SIGS styles, SIGS policies and
% procedures, Conferences etc. should be sent to
% Adrienne Griscti (griscti@acm.org)
%
% Technical questions _only_ to
% Gerald Murray (murray@hq.acm.org)
% ===============================================================
%
% For tracking purposes - this is V2.0 - May 2012

\documentclass{sig-alternate}

\begin{document}
%
% --- Author Metadata here ---
\conferenceinfo{WOODSTOCK}{'97 El Paso, Texas USA}
%\CopyrightYear{2007} % Allows default copyright year (20XX) to be over-ridden - IF NEED BE.
%\crdata{0-12345-67-8/90/01}  % Allows default copyright data (0-89791-88-6/97/05) to be over-ridden - IF NEED BE.
% --- End of Author Metadata ---

\title{Alternate {\ttlit ACM} SIG Proceedings Paper in LaTeX
Format\titlenote{(Produces the permission block, and
copyright information). For use with
SIG-ALTERNATE.CLS. Supported by ACM.}}
\subtitle{[Extended Abstract]
\titlenote{A full version of this paper is available as
\textit{Author's Guide to Preparing ACM SIG Proceedings Using
\LaTeX$2_\epsilon$\ and BibTeX} at
\texttt{www.acm.org/eaddress.htm}}}
%
% You need the command \numberofauthors to handle the 'placement
% and alignment' of the authors beneath the title.
%
% For aesthetic reasons, we recommend 'three authors at a time'
% i.e. three 'name/affiliation blocks' be placed beneath the title.
%
% NOTE: You are NOT restricted in how many 'rows' of
% "name/affiliations" may appear. We just ask that you restrict
% the number of 'columns' to three.
%
% Because of the available 'opening page real-estate'
% we ask you to refrain from putting more than six authors
% (two rows with three columns) beneath the article title.
% More than six makes the first-page appear very cluttered indeed.
%
% Use the \alignauthor commands to handle the names
% and affiliations for an 'aesthetic maximum' of six authors.
% Add names, affiliations, addresses for
% the seventh etc. author(s) as the argument for the
% \additionalauthors command.
% These 'additional authors' will be output/set for you
% without further effort on your part as the last section in
% the body of your article BEFORE References or any Appendices.

\numberofauthors{8} %  in this sample file, there are a *total*
% of EIGHT authors. SIX appear on the 'first-page' (for formatting
% reasons) and the remaining two appear in the \additionalauthors section.
%
\author{
% You can go ahead and credit any number of authors here,
% e.g. one 'row of three' or two rows (consisting of one row of three
% and a second row of one, two or three).
%
% The command \alignauthor (no curly braces needed) should
% precede each author name, affiliation/snail-mail address and
% e-mail address. Additionally, tag each line of
% affiliation/address with \affaddr, and tag the
% e-mail address with \email.
%
% 1st. author
\alignauthor
Ben Trovato\titlenote{Dr.~Trovato insisted his name be first.}\\
       \affaddr{Institute for Clarity in Documentation}\\
       \affaddr{1932 Wallamaloo Lane}\\
       \affaddr{Wallamaloo, New Zealand}\\
       \email{trovato@corporation.com}
% 2nd. author
\alignauthor
G.K.M. Tobin\titlenote{The secretary disavows
any knowledge of this author's actions.}\\
       \affaddr{Institute for Clarity in Documentation}\\
       \affaddr{P.O. Box 1212}\\
       \affaddr{Dublin, Ohio 43017-6221}\\
       \email{webmaster@marysville-ohio.com}
% 3rd. author
\alignauthor Lars Th{\o}rv{\"a}ld\titlenote{This author is the
one who did all the really hard work.}\\
       \affaddr{The Th{\o}rv{\"a}ld Group}\\
       \affaddr{1 Th{\o}rv{\"a}ld Circle}\\
       \affaddr{Hekla, Iceland}\\
       \email{larst@affiliation.org}
\and  % use '\and' if you need 'another row' of author names
% 4th. author
\alignauthor Lawrence P. Leipuner\\
       \affaddr{Brookhaven Laboratories}\\
       \affaddr{Brookhaven National Lab}\\
       \affaddr{P.O. Box 5000}\\
       \email{lleipuner@researchlabs.org}
% 5th. author
\alignauthor Sean Fogarty\\
       \affaddr{NASA Ames Research Center}\\
       \affaddr{Moffett Field}\\
       \affaddr{California 94035}\\
       \email{fogartys@amesres.org}
% 6th. author
\alignauthor Charles Palmer\\
       \affaddr{Palmer Research Laboratories}\\
       \affaddr{8600 Datapoint Drive}\\
       \affaddr{San Antonio, Texas 78229}\\
       \email{cpalmer@prl.com}
}
% There's nothing stopping you putting the seventh, eighth, etc.
% author on the opening page (as the 'third row') but we ask,
% for aesthetic reasons that you place these 'additional authors'
% in the \additional authors block, viz.
\additionalauthors{Additional authors: John Smith (The Th{\o}rv{\"a}ld Group,
email: {\texttt{jsmith@affiliation.org}}) and Julius P.~Kumquat
(The Kumquat Consortium, email: {\texttt{jpkumquat@consortium.net}}).}
\date{30 July 1999}
% Just remember to make sure that the TOTAL number of authors
% is the number that will appear on the first page PLUS the
% number that will appear in the \additionalauthors section.

\maketitle
\begin{abstract}
This paper provides a sample of a \LaTeX\ document which conforms,
somewhat loosely, to the formatting guidelines for
ACM SIG Proceedings. It is an {\em alternate} style which produces
a {\em tighter-looking} paper and was designed in response to
concerns expressed, by authors, over page-budgets.
It complements the document \textit{Author's (Alternate) Guide to
Preparing ACM SIG Proceedings Using \LaTeX$2_\epsilon$\ and Bib\TeX}.
This source file has been written with the intention of being
compiled under \LaTeX$2_\epsilon$\ and BibTeX.

The developers have tried to include every imaginable sort
of ``bells and whistles", such as a subtitle, footnotes on
title, subtitle and authors, as well as in the text, and
every optional component (e.g. Acknowledgments, Additional
Authors, Appendices), not to mention examples of
equations, theorems, tables and figures.

To make best use of this sample document, run it through \LaTeX\
and BibTeX, and compare this source code with the printed
output produced by the dvi file. A compiled PDF version
is available on the web page to help you with the
`look and feel'.
\end{abstract}

% A category with the (minimum) three required fields
\category{H.4}{Information Systems Applications}{Miscellaneous}
%A category including the fourth, optional field follows...
\category{D.2.8}{Software Engineering}{Metrics}[complexity measures, performance measures]

\terms{Theory}

\keywords{ACM proceedings, \LaTeX, text tagging}

\section{Introduction}
The \textit{proceedings} are the records of a conference.
ACM seeks to give these conference by-products a uniform,
high-quality appearance.  To do this, ACM has some rigid
requirements for the format of the proceedings documents: there
is a specified format (balanced  double columns), a specified
set of fonts (Arial or Helvetica and Times Roman) in
certain specified sizes (for instance, 9 point for body copy),
a specified live area (18 $\times$ 23.5 cm [7" $\times$ 9.25"]) centered on
the page, specified size of margins (1.9 cm [0.75"]) top, (2.54 cm [1"]) bottom
and (1.9 cm [.75"]) left and right; specified column width
(8.45 cm [3.33"]) and gutter size (.83 cm [.33"]).

The good news is, with only a handful of manual
settings\footnote{Two of these, the {\texttt{\char'134 numberofauthors}}
and {\texttt{\char'134 alignauthor}} commands, you have
already used; another, {\texttt{\char'134 balancecolumns}}, will
be used in your very last run of \LaTeX\ to ensure
balanced column heights on the last page.}, the \LaTeX\ document
class file handles all of this for you.

The remainder of this document is concerned with showing, in
the context of an ``actual'' document, the \LaTeX\ commands
specifically available for denoting the structure of a
proceedings paper, rather than with giving rigorous descriptions
or explanations of such commands.

\section{The {\secit Body} of The Paper}
Typically, the body of a paper is organized
into a hierarchical structure, with numbered or unnumbered
headings for sections, subsections, sub-subsections, and even
smaller sections.  The command \texttt{{\char'134}section} that
precedes this paragraph is part of such a
hierarchy.\footnote{This is the second footnote.  It
starts a series of three footnotes that add nothing
informational, but just give an idea of how footnotes work
and look. It is a wordy one, just so you see
how a longish one plays out.} \LaTeX\ handles the numbering
and placement of these headings for you, when you use
the appropriate heading commands around the titles
of the headings.  If you want a sub-subsection or
smaller part to be unnumbered in your output, simply append an
asterisk to the command name.  Examples of both
numbered and unnumbered headings will appear throughout the
balance of this sample document.

Because the entire article is contained in
the \textbf{document} environment, you can indicate the
start of a new paragraph with a blank line in your
input file; that is why this sentence forms a separate paragraph.

\subsection{Type Changes and {\subsecit Special} Characters}
We have already seen several typeface changes in this sample.  You
can indicate italicized words or phrases in your text with
the command \texttt{{\char'134}textit}; emboldening with the
command \texttt{{\char'134}textbf}
and typewriter-style (for instance, for computer code) with
\texttt{{\char'134}texttt}.  But remember, you do not
have to indicate typestyle changes when such changes are
part of the \textit{structural} elements of your
article; for instance, the heading of this subsection will
be in a sans serif\footnote{A third footnote, here.
Let's make this a rather short one to
see how it looks.} typeface, but that is handled by the
document class file. Take care with the use
of\footnote{A fourth, and last, footnote.}
the curly braces in typeface changes; they mark
the beginning and end of
the text that is to be in the different typeface.

You can use whatever symbols, accented characters, or
non-English characters you need anywhere in your document;
you can find a complete list of what is
available in the \textit{\LaTeX\
User's Guide}\cite{Lamport:LaTeX}.

\subsection{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of
the three are discussed in the next sections.

\subsubsection{Inline (In-text) Equations}
A formula that appears in the running text is called an
inline or in-text formula.  It is produced by the
\textbf{math} environment, which can be
invoked with the usual \texttt{{\char'134}begin. . .{\char'134}end}
construction or with the short form \texttt{\$. . .\$}. You
can use any of the symbols and structures,
from $\alpha$ to $\omega$, available in
\LaTeX\cite{Lamport:LaTeX}; this section will simply show a
few examples of in-text equations in context. Notice how
this equation: \begin{math}\lim_{n\rightarrow \infty}x=0\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsubsection{Display Equations}
A numbered display equation -- one set off by vertical space
from the text and centered horizontally -- is produced
by the \textbf{equation} environment. An unnumbered display
equation is produced by the \textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols
and structures available in \LaTeX; this section will just
give a couple of examples of display equations in context.
First, consider the equation, shown as an inline equation above:
\begin{equation}\lim_{n\rightarrow \infty}x=0\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}\sum_{i=0}^{\infty} x + 1\end{displaymath}
and follow it with another numbered equation:
\begin{equation}\sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\subsection{Citations}
Citations to articles \cite{bowman:reasoning,
clark:pct, braams:babel, herlihy:methodology},
conference proceedings \cite{clark:pct} or
books \cite{salas:calculus, Lamport:LaTeX} listed
in the Bibliography section of your
article will occur throughout the text of your article.
You should use BibTeX to automatically produce this bibliography;
you simply need to insert one of several citation commands with
a key of the item cited in the proper location in
the \texttt{.tex} file \cite{Lamport:LaTeX}.
The key is a short reference you invent to uniquely
identify each work; in this sample document, the key is
the first author's surname and a
word from the title.  This identifying key is included
with each item in the \texttt{.bib} file for your article.

The details of the construction of the \texttt{.bib} file
are beyond the scope of this sample document, but more
information can be found in the \textit{Author's Guide},
and exhaustive details in the \textit{\LaTeX\ User's
Guide}\cite{Lamport:LaTeX}.

This article shows only the plainest form
of the citation command, using \texttt{{\char'134}cite}.
This is what is stipulated in the SIGS style specifications.
No other citation format is endorsed or supported.

\subsection{Tables}
Because tables cannot be split across pages, the best
placement for them is typically the top of the page
nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and
the table caption.  The contents of the table itself must go
in the \textbf{tabular} environment, to
be aligned properly in rows and columns, with the desired
horizontal and vertical rules.  Again, detailed instructions
on \textbf{tabular} material
is found in the \textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table 1 is included in the input file; compare the
placement of the table here with the table in the printed
dvi output of this document.

\begin{table}
\centering
\caption{Frequency of Special Characters}
\begin{tabular}{|c|c|l|} \hline
Non-English or Math&Frequency&Comments\\ \hline
\O & 1 in 1,000& For Swedish names\\ \hline
$\pi$ & 1 in 5& Common in math\\ \hline
\$ & 4 in 5 & Used in business\\ \hline
$\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
\hline\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of
the page's live area, use the environment
\textbf{table*} to enclose the table's contents and
the table caption.  As with a single-column table, this wide
table will ``float" to a location deemed more desirable.
Immediately following this sentence is the point at which
Table 2 is included in the input file; again, it is
instructive to compare the placement of the
table here with the table in the printed dvi
output of this document.


\begin{table*}
\centering
\caption{Some Typical Commands}
\begin{tabular}{|c|c|l|} \hline
Command&A Number&Comments\\ \hline
\texttt{{\char'134}alignauthor} & 100& Author alignment\\ \hline
\texttt{{\char'134}numberofauthors}& 200& Author enumeration\\ \hline
\texttt{{\char'134}table}& 300 & For tables\\ \hline
\texttt{{\char'134}table*}& 400& For wider tables\\ \hline\end{tabular}
\end{table*}
% end the environment with {table*}, NOTE not {table}!

\subsection{Figures}
Like tables, figures cannot be split across pages; the
best placement for them
is typically the top or the bottom of the page nearest
their initial cite.  To ensure this proper ``floating'' placement
of figures, use the environment
\textbf{figure} to enclose the figure and its caption.

This sample document contains examples of \textbf{.eps}
and \textbf{.ps} files to be displayable with \LaTeX.  More
details on each of these is found in the \textit{Author's Guide}.

\begin{figure}
\centering
\epsfig{file=fly.eps}
\caption{A sample black and white graphic (.eps format).}
\end{figure}

\begin{figure}
\centering
\epsfig{file=fly.eps, height=1in, width=1in}
\caption{A sample black and white graphic (.eps format)
that has been resized with the \texttt{epsfig} command.}
\end{figure}


As was the case with tables, you may want a figure
that spans two columns.  To do this, and still to
ensure proper ``floating'' placement of tables, use the environment
\textbf{figure*} to enclose the figure and its caption.
and don't forget to end the environment with
{figure*}, not {figure}!

\begin{figure*}
\centering
\epsfig{file=flies.eps}
\caption{A sample black and white graphic (.eps format)
that needs to span two columns of text.}
\end{figure*}

Note that either {\textbf{.ps}} or {\textbf{.eps}} formats are
used; use
the \texttt{{\char'134}epsfig} or \texttt{{\char'134}psfig}
commands as appropriate for the different file types.

\begin{figure}
\centering
\psfig{file=rosette.ps, height=1in, width=1in,}
\caption{A sample black and white graphic (.ps format) that has
been resized with the \texttt{psfig} command.}
\vskip -6pt
\end{figure}

\subsection{Theorem-like Constructs}
Other common constructs that may occur in your article are
the forms for logical constructs like theorems, axioms,
corollaries and proofs.  There are
two forms, one produced by the
command \texttt{{\char'134}newtheorem} and the
other by the command \texttt{{\char'134}newdef}; perhaps
the clearest and easiest way to distinguish them is
to compare the two in the output of this sample document:

This uses the \textbf{theorem} environment, created by
the\linebreak\texttt{{\char'134}newtheorem} command:
\newtheorem{theorem}{Theorem}
\begin{theorem}
Let $f$ be continuous on $[a,b]$.  If $G$ is
an antiderivative for $f$ on $[a,b]$, then
\begin{displaymath}\int^b_af(t)dt = G(b) - G(a).\end{displaymath}
\end{theorem}

The other uses the \textbf{definition} environment, created
by the \texttt{{\char'134}newdef} command:
\newdef{definition}{Definition}
\begin{definition}
If $z$ is irrational, then by $e^z$ we mean the
unique number which has
logarithm $z$: \begin{displaymath}{\log e^z = z}\end{displaymath}
\end{definition}

Two lists of constructs that use one of these
forms is given in the
\textit{Author's  Guidelines}.
 
There is one other similar construct environment, which is
already set up
for you; i.e. you must \textit{not} use
a \texttt{{\char'134}newdef} command to
create it: the \textbf{proof} environment.  Here
is a example of its use:
\begin{proof}
Suppose on the contrary there exists a real number $L$ such that
\begin{displaymath}
\lim_{x\rightarrow\infty} \frac{f(x)}{g(x)} = L.
\end{displaymath}
Then
\begin{displaymath}
l=\lim_{x\rightarrow c} f(x)
= \lim_{x\rightarrow c}
\left[ g{x} \cdot \frac{f(x)}{g(x)} \right ]
= \lim_{x\rightarrow c} g(x) \cdot \lim_{x\rightarrow c}
\frac{f(x)}{g(x)} = 0\cdot L = 0,
\end{displaymath}
which contradicts our assumption that $l\neq 0$.
\end{proof}

Complete rules about using these environments and using the
two different creation commands are in the
\textit{Author's Guide}; please consult it for more
detailed instructions.  If you need to use another construct,
not listed therein, which you want to have the same
formatting as the Theorem
or the Definition\cite{salas:calculus} shown above,
use the \texttt{{\char'134}newtheorem} or the
\texttt{{\char'134}newdef} command,
respectively, to create it.

\subsection*{A {\secit Caveat} for the \TeX\ Expert}
Because you have just been given permission to
use the \texttt{{\char'134}newdef} command to create a
new form, you might think you can
use \TeX's \texttt{{\char'134}def} to create a
new command: \textit{Please refrain from doing this!}
Remember that your \LaTeX\ source code is primarily intended
to create camera-ready copy, but may be converted
to other forms -- e.g. HTML. If you inadvertently omit
some or all of the \texttt{{\char'134}def}s recompilation will
be, to say the least, problematic.

\section{Conclusions}
This paragraph will end the body of this sample document.
Remember that you might still have Acknowledgments or
Appendices; brief samples of these
follow.  There is still the Bibliography to deal with; and
we will make a disclaimer about that here: with the exception
of the reference to the \LaTeX\ book, the citations in
this paper are to articles which have nothing to
do with the present subject and are used as
examples only.
%\end{document}  % This is where a 'short' article might terminate

%ACKNOWLEDGMENTS are optional
\section{Acknowledgments}
This section is optional; it is a location for you
to acknowledge grants, funding, editing assistance and
what have you.  In the present case, for example, the
authors would like to thank Gerald Murray of ACM for
his help in codifying this \textit{Author's Guide}
and the \textbf{.cls} and \textbf{.tex} files that it describes.

%
% The following two commands are all you need in the
% initial runs of your .tex file to
% produce the bibliography for the citations in your paper.
\bibliographystyle{abbrv}
\bibliography{sigproc}  % sigproc.bib is the name of the Bibliography in this case
% You must have a proper ".bib" file
%  and remember to run:
% latex bibtex latex latex
% to resolve all references
%
% ACM needs 'a single self-contained file'!
%
%APPENDICES are optional
%\balancecolumns
\appendix
%Appendix A
\section{Headings in Appendices}
The rules about hierarchical headings discussed above for
the body of the article are different in the appendices.
In the \textbf{appendix} environment, the command
\textbf{section} is used to
indicate the start of each Appendix, with alphabetic order
designation (i.e. the first is A, the second B, etc.) and
a title (if you include one).  So, if you need
hierarchical structure
\textit{within} an Appendix, start with \textbf{subsection} as the
highest level. Here is an outline of the body of this
document in Appendix-appropriate form:
\subsection{Introduction}
\subsection{The Body of the Paper}
\subsubsection{Type Changes and  Special Characters}
\subsubsection{Math Equations}
\paragraph{Inline (In-text) Equations}
\paragraph{Display Equations}
\subsubsection{Citations}
\subsubsection{Tables}
\subsubsection{Figures}
\subsubsection{Theorem-like Constructs}
\subsubsection*{A Caveat for the \TeX\ Expert}
\subsection{Conclusions}
\subsection{Acknowledgments}
\subsection{Additional Authors}
This section is inserted by \LaTeX; you do not insert it.
You just add the names and information in the
\texttt{{\char'134}additionalauthors} command at the start
of the document.
\subsection{References}
Generated by bibtex from your ~.bib file.  Run latex,
then bibtex, then latex twice (to resolve references)
to create the ~.bbl file.  Insert that ~.bbl file into
the .tex source file and comment out
the command \texttt{{\char'134}thebibliography}.
% This next section command marks the start of
% Appendix B, and does not continue the present hierarchy
\section{More Help for the Hardy}
The sig-alternate.cls file itself is chock-full of succinct
and helpful comments.  If you consider yourself a moderately
experienced to expert user of \LaTeX, you may find reading
it useful but please remember not to change it.
%\balancecolumns % GM June 2007
% That's all folks!
\end{document}

Validity.tex
% We have made some assumptions in the implementation and the empirical design, which we describe below and we also talk about how we manage to mitigate the validity threats associated
% with them.
%We describe some assumptions in the paper and discuss how we manage to mitigate the validity threats associated with them.

%User study has been users to substantiate our algorithm effectiveness in improving user satisfaction by highlighting most relevant passages in the research papers. 
%The following assumptions have been made in the implementation and our user study was able to prove that these assumptions do not hinder the performance of our algorithm. 
%First of all, the 
% composed of Abstract, Introduction, Related Work, Implementation, Evaluation, and Future Work/Conclusion. 
We have assumed a fix set of target sections to match, training our system entirely on examples from the Software Engineering domain. We have assumed test documents will be similar to training documents. While our participants evaluated our system on papers from wider Information and Computer Science domains, our current system remains very far from being able to support all scholarly genres and document formatting and structure. Our evaluation shows in the case of mismatch between training and testing documents used here, Bullseye appears to exhibit at least comparable performance to simply ignoring document structure, as done in the baseline system, given our fallback design. 


Another issue is that the decision trees used by the Bullseye reflect our own empirical inspection of the training set. While we also applied two automatic decision tree induction techniques~\cite{umanol1994fuzzy}~\cite{ruggieri2002efficient}, the trees we constructed ourselves proved more effective in practice. That said, future work should develop an effective automated method to reduce manual effort and strive for greater robustness and generality over a wider range of scholarly domains. 

%Though we have attempted to use automatic decision tree induction process, 
%we found the decision trees based on empirical study is much simpler to implement and more appropriate for our research scenarios than the two decision tree induction techniques~\cite{umanol1994fuzzy}~\cite{ruggieri2002efficient}  tried. 
%It might be possible other automatic tools for decision trees
%or other machine learning techniques are more suitable to locate section headers, we argue due to the fixed set of the section headers we defined for the search condition, the current decision trees generated
%shall be sufficient.

%First of all the implicit section location algorithm of Bullseye is based on manual analysis of the training set. The derived rules are implemented in the form of decision trees.
%This could result in some exceptions to the decision trees introduced by some other documents not included in the training set. However..... 
%Last validity threat is the bias in creating the test collection as the test collection might be not representative enough of general research papers.  To mitigate this threat, we asked the participants from different domain to prepare their own topics and create gold true passages themselves.

% Finally to avoid users giving us a biased opinion to the Bullseye algorithm, we alternated the document name for the Baseline and the Bullseye so as to ensure that the participants
% had no idea of the correlation between documents and algorithms.  We gave each user six test suites, including one from the user himself, to avoid learning effect.

BullsEye.bbl
\begin{thebibliography}{10}

\bibitem{al2010review}
Azzah Al-Maskari and Mark Sanderson.
\newblock A review of factors influencing user satisfaction in information
  retrieval.
\newblock {\em Journal of the American Society for Information Science and
  Technology}, 61(5):859--868, 2010.

\bibitem{allan2005will}
James Allan, Ben Carterette, and Joshua Lewis.
\newblock When will information retrieval be good enough?
\newblock In {\em Proceedings of the 28th annual international ACM SIGIR
  conference on Research and development in information retrieval}, pages
  433--440. ACM, 2005.

\bibitem{croft1994corpus}
Bruce Croft and Jinxi Xu.
\newblock {\em Corpus-specific stemming using word form co-occurence}.
\newblock Citeseer, 1994.

\bibitem{denny2009evaluation}
Joshua~C Denny, Anderson Spickard~III, Kevin~B Johnson, Neeraja~B Peterson,
  Josh~F Peterson, and Randolph~A Miller.
\newblock Evaluation of a method to identify and categorize section headers in
  clinical documents.
\newblock {\em Journal of the American Medical Informatics Association},
  16(6):806--815, 2009.

\bibitem{frank2000text}
Eibe Frank, Chang Chui, and Ian~H Witten.
\newblock Text categorization using compression models.
\newblock 2000.

\bibitem{kamps2003xml}
Jaap Kamps, Maarten Marx, Maarten De~Rijke, and B{\"o}rkur Sigurbj{\"o}rnsson.
\newblock Xml retrieval: What to retrieve?
\newblock In {\em Proceedings of the 26th annual international ACM SIGIR
  conference on Research and development in informaion retrieval}, pages
  409--410. ACM, 2003.

\bibitem{kaszkiel1997passage}
Marcin Kaszkiel and Justin Zobel.
\newblock Passage retrieval revisited.
\newblock In {\em ACM SIGIR Forum}, pages 178--185, 1997.

\bibitem{knaus1996highlighting}
Daniel Knaus, Elke Mittendorf, Peter Schauble, and Paraic Sheridan.
\newblock Highlighting relevant passages for users of the interactive spider
  retrieval system.
\newblock In {\em Proceedings of the fourth text retrieval conference
  (TREC-4)}, pages 233--244, 1996.

\bibitem{lalmas2007evaluating}
Mounia Lalmas and Anastasios Tombros.
\newblock Evaluating xml retrieval effectiveness at inex.
\newblock In {\em SIGIR Forum}, pages 40--57, 2007.

\bibitem{light2001analyses}
Marc Light, Gideon~S Mann, Ellen Riloff, and Eric Breck.
\newblock Analyses for elucidating current question answering technology.
\newblock {\em Natural Language Engineering}, 7(04):325--342, 2001.

\bibitem{milic2005facility}
Natasa Milic-Frayling and Ralph Sommerer.
\newblock Facility for highlighting documents accessed through search or
  browsing, November~22 2005.
\newblock US Patent 6,968,332.

\bibitem{PDFBox}
{PDFBox}.
\newblock \url{http://pdfbox.apache.org/index.html}.

\bibitem{ruggieri2002efficient}
Salvatore Ruggieri.
\newblock Efficient c4. 5 [classification algorithm].
\newblock {\em Knowledge and Data Engineering, IEEE Transactions on},
  14(2):438--444, 2002.

\bibitem{salton1993approaches}
Gerard Salton, James Allan, and Chris Buckley.
\newblock Approaches to passage retrieval in full text information systems.
\newblock In {\em Proceedings of the 16th annual international ACM SIGIR
  conference on Research and development in information retrieval}, pages
  49--58. ACM, 1993.

\bibitem{trotman2007report}
Andrew Trotman, Shlomo Geva, and Jaap Kamps.
\newblock Report on the sigir 2007 workshop on focused retrieval.
\newblock In {\em ACM SIGIR Forum}, pages 97--103, 2007.

\bibitem{umanol1994fuzzy}
M~Umanol, Hirotaka Okamoto, Itsuo Hatono, HIROYUKI Tamura, Fumio Kawachi,
  Sukehisa Umedzu, and Junichi Kinoshita.
\newblock Fuzzy decision trees by fuzzy id3 algorithm and its application to
  diagnosis systems.
\newblock In {\em Fuzzy Systems, 1994. IEEE World Congress on Computational
  Intelligence., Proceedings of the Third IEEE Conference on}, pages
  2113--2118. IEEE, 1994.

\end{thebibliography}

EvaluateDocumentRetrievalPassageRetrieval.bbl
\begin{thebibliography}{1}

\bibitem{al2010review}
Azzah Al-Maskari and Mark Sanderson.
\newblock A review of factors influencing user satisfaction in information
  retrieval.
\newblock {\em Journal of the American Society for Information Science and
  Technology}, 61(5):859--868, 2010.

\bibitem{cui2003hierarchical}
Hang Cui, Ji-Rong Wen, and Tat-Seng Chua.
\newblock Hierarchical indexing and flexible element retrieval for structured
  document.
\newblock In {\em Advances in Information Retrieval}, pages 73--87. Springer,
  2003.

\bibitem{denny2009evaluation}
Joshua~C Denny, Anderson Spickard~III, Kevin~B Johnson, Neeraja~B Peterson,
  Josh~F Peterson, and Randolph~A Miller.
\newblock Evaluation of a method to identify and categorize section headers in
  clinical documents.
\newblock {\em Journal of the American Medical Informatics Association},
  16(6):806--815, 2009.

\bibitem{kamps2003xml}
Jaap Kamps, Maarten Marx, Maarten De~Rijke, and B{\"o}rkur Sigurbj{\"o}rnsson.
\newblock Xml retrieval: What to retrieve?
\newblock In {\em Proceedings of the 26th annual international ACM SIGIR
  conference on Research and development in informaion retrieval}, pages
  409--410. ACM, 2003.

\bibitem{lalmas2007evaluating}
Mounia Lalmas and Anastasios Tombros.
\newblock Evaluating xml retrieval effectiveness at inex.
\newblock In {\em ACM SIGIR Forum}, volume~41, pages 40--57. ACM, 2007.

\bibitem{light2001analyses}
Marc Light, Gideon~S Mann, Ellen Riloff, and Eric Breck.
\newblock Analyses for elucidating current question answering technology.
\newblock {\em Natural Language Engineering}, 7(04):325--342, 2001.

\bibitem{singhal1996document}
Amit Singhal, Gerard Salton, Mandar Mitra, and Chris Buckley.
\newblock Document length normalization.
\newblock {\em Information Processing \& Management}, 32(5):619--633, 1996.

\bibitem{tellex2003quantitative}
Stefanie Tellex, Boris Katz, Jimmy Lin, Aaron Fernandes, and Gregory Marton.
\newblock Quantitative evaluation of passage retrieval algorithms for question
  answering.
\newblock In {\em Proceedings of the 26th annual international ACM SIGIR
  conference on Research and development in informaion retrieval}, pages
  41--47. ACM, 2003.

\end{thebibliography}
