Two Training Strategies for Improving Relation Extraction over Universal Graph
Qin Dai$^1$
Naoya Inoue$^{2
This paper explores how the Distantly Supervised Relation Extraction (DS-RE) can benefit from the use of a Universal Graph (UG), the combination of a Knowledge Graph (KG) and a large-scale text collection. A straightforward extension of a current state-of-the-art neural model for DS-RE with a UG may lead to degradation in performance. We first report that this degradation is associated with the difficulty in learning a UG and then propose two training strategies: (1) Path Type Adaptive Pretraining, which sequentially trains the model with different types of UG paths so as to prevent the reliance on a single type of UG path; and (2) Complexity Ranking Guided Attention mechanism, which restricts the attention span according to the complexity of a UG path so as to force the model to extract features not only from simple UG paths but also from complex ones. Experimental results on both biomedical and NYT10 datasets prove the robustness of our methods and achieve a new state-of-the-art result on the NYT10 dataset. The code and datasets used in this paper are available at \url{https://github.com/baodaiqin/UGDSRE.
eacl_2021.tex
%
% File coling2020.tex
%
% Contact: feiliu@cs.ucf.edu & liang.huang.sh@gmail.com
%% Based on the style files for COLING-2018, which were, in turn,
%% Based on the style files for COLING-2016, which were, in turn,
%% Based on the style files for COLING-2014, which were, in turn,
%% Based on the style files for ACL-2014, which were, in turn,
%% Based on the style files for ACL-2013, which were, in turn,
%% Based on the style files for ACL-2012, which were, in turn,
%% based on the style files for ACL-2011, which were, in turn, 
%% based on the style files for ACL-2010, which were, in turn, 
%% based on the style files for ACL-IJCNLP-2009, which were, in turn,
%% based on the style files for EACL-2009 and IJCNLP-2008...

%% Based on the style files for EACL 2006 by 
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith

\documentclass[11pt,a4paper]{article}
\usepackage[hyperref]{eacl2021}
%\usepackage{eacl2021}
\usepackage{times}
\usepackage{url}

\usepackage{latexsym}
\renewcommand{\UrlFont}{\ttfamily\small}
\usepackage{microtype}

\usepackage{times}
\usepackage{latexsym}
\usepackage{lingmacros}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{array, tabularx, caption, boldline}
\usepackage{cellspace}
\usepackage{url}
\usepackage{makecell}
\usepackage{multirow}
\usepackage{xcolor}

\usepackage{subcaption}
\usepackage{booktabs}
\usepackage{comment}

\usepackage{pifont}% http://ctan.org/pkg/pifont
\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%


%\setlength\titlebox{5cm}
\aclfinalcopy % Uncomment this line for the final submission

% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.

%\newcommand\BibTeX{B\textsc{ib}\TeX}
%https://www.overleaf.com/project/5f770bf104280b00019caff6
\title{Two Training Strategies for Improving Relation Extraction over Universal Graph}

\author{Qin Dai$^1$, Naoya Inoue$^{2}$, Ryo Takahashi$^{1,3}$, Kentaro Inui$^{1,3}$ \\ $^1$Tohoku University, Japan \\ $^2$Stony Brook University, USA \\ $^3$RIKEN Center for Advanced Intelligence Project, Japan \\ {\tt \{daiqin, naoya-i, ryo.t, inui\}@ecei.tohoku.ac.jp}}

\date{}

\begin{document}
\maketitle
\begin{abstract}
   This paper explores how the Distantly Supervised Relation Extraction (DS-RE) can benefit from the use of a Universal Graph (UG), the combination of a Knowledge Graph (KG) and a large-scale text collection. A straightforward extension of a current state-of-the-art neural model for DS-RE with a UG may lead to degradation in performance. We first report that this degradation is associated with the difficulty in learning a UG and then propose two training strategies: (1) Path Type Adaptive Pretraining, which sequentially trains the model with different types of UG paths so as to prevent the reliance on a single type of UG path; and (2) Complexity Ranking Guided Attention mechanism, which restricts the attention span according to the complexity of a UG path so as to force the model to extract features not only from simple UG paths but also from complex ones. Experimental results on both biomedical and NYT10 datasets prove the robustness of our methods and achieve a new state-of-the-art result on the NYT10 dataset. The code and datasets used in this paper are available at \url{https://github.com/baodaiqin/UGDSRE}.

\end{abstract}

\section{Introduction}
\label{sec:intro}
Relation Extraction (RE) is an important task in Natural Language Processing (NLP). RE aims to turn unstructured texts into structured Knowledge Graph (KG), which is typically stored as ($e_1$, $r$, $e_2$) triplets, where $e_1$ is a \textit{head entity}, $r$ is a relation and $e_2$ is a \textit{tail entity}, such as (\textit{aspirin}, \textit{may\_treat}, \textit{pain}) and (\textit{Guy Maddin}, \textit{place\_lived}, \textit{Winnpeg}). RE can be formulated as a classification task to predict a predefined relation $r$ from entity pair $(e_1,e_2)$ annotated evidences.

One obstacle that is encountered when building a RE system is the generation of a large amount of manually annotated training instances, which is expensive and time-consuming. For coping with this difficulty, \newcite{mintz2009distant} propose Distant Supervision (DS) to automatically generate training samples via linking KGs to texts. They assume that if ($e_1$, $r$, $e_2$) is in a KG, then all sentences that contain ($e_1$, $e_2$) (hereafter, \emph{sentence evidences}) express the relation $r$. It is well known that the DS assumption is too strong and inevitably accompanies the wrong labeling problem, such as the sentence evidences (\ref{sent:wronglabel} and \ref{sent:wronglabel_gen}) below, which fail to express \textit{may\_treat} and \textit{place\_lived} relation respectively.
%
\eenumsentence{
    \item {\it \textbf{Aspirin}$_{e_1}$ is widely used for short-term treatment of \textbf{pain}$_{e_2}$, fever or colds.} \label{sent:goodlabel}
    \item {\it The tumor was remarkably large in size, and \textbf{pain}$_{e_2}$ unrelieved by \textbf{aspirin}$_{e_1}$.} \label{sent:wronglabel}
}
\enumsentence{
    {\it He is now finishing a documentary about \textbf{Winnipeg}$_{e_2}$, the final installment of a personal trilogy that began with “Cowards Bend the Knee” (a 2003 film that also featured a hapless hero named \textbf{Guy Maddin}$_{e_1}$).}
    \label{sent:wronglabel_gen}
}

Recently, neural network models with attention mechanism have been proposed to alleviate the wrong labeling problem and attend informative sentence evidences such as (\ref{sent:goodlabel})~\cite{lin2016neural,ji2017distant,du2018multi,jat2018improving,han2018neural,han2018hierarchical}.
However, there can be a large portion of entity pairs that lack such informative sentence evidences that explicitly express their relation.
This makes Distantly Supervised Relation Extraction (DS-RE) further challenging~\cite{sun2019leveraging}.
%
\begin{figure*}[t]
\centering
% \includegraphics[height=8.0cm, width=\columnwidth*2]{graph/eacl_intro.pdf}
\includegraphics[width=\columnwidth*2]{graph/eacl_intro.pdf}
\caption{Overview of our UG-based framework, where \textbf{Colesevelam HCl} and \textbf{Type 2 Diabetes} are the target entities, \textbf{COLESEVELAM ...}, \textbf{Colestipol} and \textbf{hyperglyceridemia} are intermediate entities, each UG path consists of multiple hops and each hop represents a KG relation (such as ``\textit{Colestipol may treat hyperglyceridemia}'') or Text (or Textual) relation (such as \textsf{TR1} and \textsf{TR2}), which is the sentence containing two (target or intermediate) entities.}
\label{fig:ugexam}
\end{figure*}
%

For compensating the lack of informative sentence evidences, \newcite{dai2019incorporating} utilize multi-hop paths connecting a target entity pair (hereafter, \emph{path}) over a KG as extra evidences for DS-RE.
An example of such multi-hop KG path can be seen in Figure~\ref{fig:ugexam}, where $p_1$ depicts a multi-hop KG path of the form of {\it $e_1$ $\underrightarrow{\textit{component\_of}}$  $e_3$ $\underrightarrow{\textit{may\_treat}}$ $e_2$}.
The model of \newcite{dai2019incorporating} uses such multi-hop paths as additional features for predicting the relation between a given target entity pair ($e_1$, $e_2$), which is reported effective for performance improvement.
However, KGs are often highly incomplete~\cite{min2013distant} and may be too sparse to provide enough informative paths in practice, which may hamper the effectiveness of multi-hop paths. 

Given this background, in this study, we take one step further, aiming for inducing maximal signals of distant supervision from both a KG and a large text collection (hereafter, \emph{Text}).
For this purpose, we consider using multi-hop paths over a Universal Graph (UG) as extra features for DS-RE. 
Here, we define a UG as a joint graph representation of both KG and Text, where each node represents an entity from KG or Text, and each edge indicates a KG relation or Textual relation, as shown in Figure~\ref{fig:ugexam}. 
The path $p_2$ in the figure is an example of UG path, comprising a textual edge \textsf{TR1}, a KG edge \textit{may\_treat}, and another textual edge \textsf{TR2}. 
By augmenting the original KG with textual edges, one can expect far more chances to find informative path evidences between any given target entity pairs, because the number of such textual edges is likely to be much larger than the number of KG edges (Note that one can collect as many textual edges as needed from a raw text corpus with an entity linker).
Extending a KG to a UG, therefore, may allow a DS-RE model to learn richer distant supervision signals. 

The idea of using multi-hop paths over a UG is not necessarily new on its own. 
For example, \newcite{toutanova2015representing} propose to use a UG for knowledge graph completion, and \newcite{das2017question} propose a model trained to reason over a UG for question answering. 
However, there is no prior study that has explored the effective way to use a UG for the task of DS-RE from text. 
In fact, finding an effective way of using a UG for DS-RE is not as simple as it may seem. 
As we report in this paper, a straightforward extension of the \newcite{dai2019incorporating} model to the UG setting may result in performance degradation. 

Motivated by this, in this paper, we address how one can make effective use of UG for DS-RE. 
We first report our observation that a straightforward extension of the \newcite{dai2019incorporating} model to the UG setting tends to allocate the majority of attention to only a limited set of UG paths such as short KG paths and miss out the learning from a wide range of UG paths (\S\ref{sec:attbia}), which hinders performance gain.
In order to alleviate the negative effect of the attention bias and realize the potential of UG paths, we propose two training  (or debiasing) strategies: (1) Path Type Adaptive Pretraining (\S\ref{sec:pre}), which aims to improve the adaptability of the model to various UG paths; and (2) Complexity Ranking Guided Attention mechanism (\S\ref{sec:rank}), which enables the model to learn from both simple and complex UG paths. Experimental results on both biomedical and NYT10~\cite{riedel2010modeling} datasets prove that: (1) UG paths have the potential to bring performance gain for DS-RE as compared with KG paths; (2) the proposed training methods are effective to fully exploit the potential of UG paths for DS-RE because the proposed methods significantly and consistently outperform several baselines on both datasets and especially achieve a new state-of-the-art result on the NYT10 dataset.



\section{Related Work}

To improve the performance of a DS-RE model, recently, researchers introduce various attention mechanisms. \newcite{lin2016neural} propose a relation vector based attention mechanism. \newcite{jat2018improving,du2018multi} propose multi-level (e.g., word-level and sentence-level) structured attention mechanism. \newcite{ye2019distant} apply both intra-bag and inter-bag attention for DS-RE. \newcite{han2018hierarchical} propose a relation hierarchy based attention mechanism. \newcite{han2018neural} propose a joint model that adopts a KG embeddings based attention mechanism. \newcite{jia2019arnor} propose an attention regularization framework to select informative sentence evidences for DS-RE. However, these models rely only on noisy sentence evidences from DS, neglecting the rich UG paths for DS-RE.

Besides the sentence evidences from DS, researchers also leverage external evidences for DS-RE. 
%\newcite{roller2015improving} use the inference pattern learned from KG for eliminating potentially related entity pairs from negative samples. \newcite{liu2017soft} propose a soft-label method that generates new training instances by dynamically correct wrong labels of training instances. 
\newcite{ji2017distant} apply entity descriptions generated from Freebase and Wikipedia as extra evidences, \newcite{lin2017neural} utilize multilingual text as extra evidences and \newcite{vashishth2018reside} use multiple extra evidences including entity types, dependency and relation alias information for DS-RE. \newcite{alt2019fine} utilize pretrained language model as background information for DS-RE. \newcite{sun2019leveraging} apply relational table extracted from Web as supplementary evidences for DS-RE. 

To apply DS-RE beyond sentence boundary, \newcite{quirk2017distant} utilize syntactic information to extract relation from neighboring sentences. \newcite{zeng2017incorporating} apply two-hop KG paths
%(e.g., \emph{/location/birthplace}, \emph{location/contains}) 
identified from two-hop textual paths as extra evidences for DS-RE. Different from this work, we directly use the rich UG paths as extra evidences.
\newcite{dai2019incorporating} extend the framework of \newcite{han2018neural} by introducing multiple KG paths as extra evidences for DS-RE. \newcite{neelakantan-etal-2015-compositional,das2017chains} use multiple reasoning paths over Text and KG for relation prediction in the paradigm of Knowledge Graph Completion. Our work differs from the ones mentioned above in two ways: (i) We utilize the UG paths as extra evidences for the task of DS-RE from text, (ii) We take into account the factor of attention bias while encoding UG paths and propose two effective debiasing methods to exploit the potential of UG paths for DS-RE.


\section{Base Model}
\label{sec:evencoder} 
We select the DS-RE model proposed by \newcite{dai2019incorporating} as our base model and extend it into our UG setting. Given a target entity pair $(e_1, e_2)$, a bag of corresponding sentence evidences $S_r=\{s_1,...,s_n\}$ and a bag of UG paths $P_r=\{p_1,...,p_m\}$, the base model aims to measure the probability of $(e_1, e_2)$ having a predefined relation $r$ (including the empty relation NA).
The base model consists of four main modules: KG Encoder, Sentence Evidence Encoder, Path Evidence Encoder and Relation Classification Layer, as shown in Figure~\ref{fig:proposed_model}.

\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{graph/eacl_rank.pdf}
\caption{Overview of the base model and our proposed Complexity Ranking Guided Attention mechanism (\S\ref{sec:rank}). The base model takes the sentence evidences (e.g., $s_1$, ...) containing a target entity pair and the UG paths (e.g., $p_1$, ...) connecting the entity pair as input for predicting their relation. The KG embeddings of the entity pair (i.e., $\mathbf{h}$ and $\mathbf{t}$) are used for calculating the attention over these sentences and paths. The Complexity Ranking Guided Attention mechanism is proposed to force the model to attend both simple UG paths (e.g., $p_1 \sim p_j$) and complex ones (e.g., $p_{m-j+1} \sim p_m$).}
\label{fig:proposed_model}
\end{figure}

\subsection{KG Encoder} Suppose we have a KG containing a set of fact triplets $\mathcal{O}=\{(e_1, r, e_2), ...\}$, where each fact triplet consists of two entities $e_1, e_2 \in \mathcal{E}$ and their relation $r \in \mathcal{R}$. Here $\mathcal{E}$ and $\mathcal{R}$ stand for the set of entities and relations respectively.

The KG Encoder then encodes $e_1, e_2 \in \mathcal{E}$ and their relation $r\in\mathcal{R}$ into low-dimensional vectors $\mathbf{h}$, $\mathbf{t}$ $\in R^d$ and $\mathbf{r}$ $\in R^d$ respectively, where $d$ is the dimensionality of the embedding space. The KG Encoder adopts TransE~\cite{bordes2013translating} to score a given triplet. Specifically, given a triplet $(e_1, r, e_2)$, TransE evaluates its plausibility via Equation~\ref{eq:kgc2}:
%
\begin{align}
f_r(e_1,e_2) & = b - \|\mathbf{r}_{ht}-\mathbf{r}\|, \label{eq:kgc2} \\
\mathbf{r}_{ht} & = \mathbf{t} - \mathbf{h}, \label{eq:transe1}
\end{align}
%
where $b$ is a bias constant and $\mathbf{r}_{ht}$ is a latent relation embedding for $(e_1, e_2)$.
The conditional probability can be formalized over all fact triplets $\mathcal{O}$ as follows:
%
\begin{equation}
\begin{split}
\mathit{P}(e_1,r,e_2|\theta_{\mathcal{E}}, \theta_{\mathcal{R}}) =
\frac{\exp(f_{r}(e_1,e_2))}{\sum_{r'\in \mathcal{R}}\exp(f_{r'}(e_1,e_2))}
\end{split}
\end{equation}
%
where $\theta_{\mathcal{E}}$ and $\theta_{\mathcal{R}}$ are parameters for entities and relations respectively. 

\subsection{Sentence Evidence Encoder} Given a bag of sentence evidences $S_r=\{s_1,...,s_n\}$, the Sentence Evidence Encoder applies CNN-Max (see Appendix \S\ref{ap:cnnmax}) on each sentence, namely $\mathbf{s}_i = \text{CNN-Max}(s_i)$, to derive the sentence representations $\{\mathbf{s}_1,...,\mathbf{s}_n\}$.
The encoder then calculates the bag-level vector representation $\mathbf{s}_{\mathrm{all}}$ via Equation~\ref{eq:cnnall}:
%
\begin{align}\label{eq:cnnall}
\mathbf{s}_{\mathrm{all}} &= \sum^n_{i=1}a_i\mathbf{s}_i,\\ 
a_i &=\frac{\exp(\langle\mathbf{r}_{ht},\mathbf{x}_i\rangle)}
{\sum^n_{k=1}\exp(\langle\mathbf{r}_{ht},\mathbf{x}_k\rangle)},\nonumber\\
\mathbf{x}_i &= \tanh(\mathbf{W}\mathbf{s}_i + \mathbf{b}) \nonumber
\end{align}
%
where $\mathbf{r}_{ht}$ is from Equation~\ref{eq:transe1}, $\mathbf{W}$ is the weight matrix, $\mathbf{b}$ is the bias vector, $a_i$ is the weight for the $i$-th sentence in $S_r$.

\subsection{Path Evidence Encoder} Given a bag of UG paths $P_r=\{p_1,...,p_m\}$ connecting an entity pair of interest ($e_1$, $e_2$), the Path Evidence Encoder encodes them into a bag-level vector representation $\mathbf{p}_{\mathrm{all}}$. Since we represent a path as a sequence of words (or a long sentence), as shown in Figure~\ref{fig:ugexam}, analogously to the Sentence Evidence Encoder, we apply a CNN-Max (see Appendix \S\ref{ap:cnnmax}) to encode each path $p_i$, namely $\mathbf{p}_i = \text{CNN-Max}(p_i)$. The bag-level path representation $\mathbf{p}_{\mathrm{all}}$ for $P_r$ is then calculated via Equation~\ref{eq:pathall}:
%
\begin{align}\label{eq:pathall}
\mathbf{p}_{\mathrm{all}} &=\sum^m_{i=1}a'_i\mathbf{p}_i,\\
a'_i &= \frac{\exp(\langle\mathbf{r}_{ht},\mathbf{x'}_i\rangle)}
{\sum^m_{k=1}\exp(\langle\mathbf{r}_{ht},\mathbf{x'}_k\rangle)},\nonumber\\
\mathbf{x'}_i &= \tanh(\mathbf{W}\mathbf{p}_i + \mathbf{b}) \nonumber
\end{align}
%
where $a'_i$ is the weight for the $i$-th path in $P_r$.

\subsection{Relation Classification Layer} The conditional probability of $(e_1,e_2)$ having a relation $r$ is formulated via Equation~\ref{eq:pathall11}:
%
\begin{equation}\label{eq:pathall11}
P(e_1,r,e_2|S_r,P_r,\theta_S, \theta_P)=\frac{\exp([\mathbf{o}]_r)}
{\sum_{c=1}^{n_r}\exp([\mathbf{o}]_c)}
\end{equation}
%
%
%\begin{equation}\label{eq:pathall12}
%\mathbf{o}=\mathbf{M}[\mathbf{s}_{\mathrm{all}}; %\mathbf{p}_{\mathrm{all}}]+\mathbf{d},
%\end{equation}
%
where $\mathbf{o}=\mathbf{M}[\mathbf{s}_{\mathrm{all}}; \mathbf{p}_{\mathrm{all}}]+\mathbf{d}$, $\theta_S$, $\theta_P$ are the parameters in Sentence Evidence Encoder and Path Evidence Encoder, $\mathbf{M}$ is the representation matrix of relations, $\mathbf{d}$ is a bias vector, $\mathbf{o}$ is the output vector containing the prediction scores of all predefined relations, $[\mathbf{o}]_c$ is the prediction score for the relation $c$, and $n_r$ is the total number of relations.

Given a training dataset consisting of triplets $\mathcal{O}=\{(e_1^1, r^1, e_2^1), (e_1^2, r^2, e_2^2), ...\}$, we minimize the objective function as follows:
%
\begin{equation}\label{eq:objective_prop}
\begin{aligned}
J(\theta)=-\frac{1}{|\mathcal{O}|}\sum_{i=1}^{|\mathcal{O}|}\log P(e_1^i,r^i,e_2^i|\theta_{\mathcal{E}},\theta_{\mathcal{R}}) \\ + \log P(e_1^i,r^i,e_2^i|S_{r^i}, P_{r^i}, \theta_S,\theta_P)
\end{aligned}
\end{equation}
%
The base model is optimized with Stochastic Gradient Descent (SGD). Following \cite{han2018neural}, we optimize $P(e_1^i,r^i,e_2^i|\theta_{\mathcal{E}},\theta_{\mathcal{R}})$ and $P(e_1^i,r^i,e_2^i|S_i, P_i, \theta_S,\theta_P)$ in parallel.


\section{Proposed Training Method}
\label{sec:proposed}

\subsection{Problem of Attention Bias}
\label{sec:attbia}


\begin{figure*}
\centering
\begin{subfigure}{\columnwidth}
  \centering
  \includegraphics[width=\columnwidth]{graph/att_pathtype.pdf}
  \caption{Path type and attention}
  \label{fig:attpathtp}
\end{subfigure}
\begin{subfigure}{\columnwidth}
  \centering
  \includegraphics[width=\columnwidth]{graph/att_length.pdf}
  \caption{Path complexity and attention}
  \label{fig:attlen}
\end{subfigure}
\caption{(\textbf{Left}) Average attention weights across different path types. (\textbf{Right}) Average attention weights across different length range of paths. Here, we use path length as an indicator of path complexity, which denotes the number of words in a path.}
\label{fig:test}
\end{figure*}

While extending the base model into our UG setting, we observe that the base model tends to allocate more attention to KG or linguistically simple paths as compared to Textual paths (i.e., the path comes form Text), Hybrid paths (i.e., the path comes from both Text and KG), or linguistically complex ones, as shown in Figure~\ref{fig:attpathtp} and Figure~\ref{fig:attlen}. We consider that this would be because paths including Textual relations (i.e., Textual and Hybrid paths) or complex paths are comparatively noisier than KG or simple paths, but which does not necessarily mean the former is not useful. For instance, in Figure~\ref{fig:ugexam}, the complex Hybrid path $p_2$ is useful for predicting (\textbf{Colesevelam\_HCl}, \emph{may\_treat}, \textbf{Type\_2\_Diabetes}), because $p_2$ implies a plausible line of reasoning `` {\it \textbf{Colesevelam\_HCl} $\underrightarrow{alternative\_to}$ Colestipol $\underrightarrow{may\_treat}$ hyperglyceridemia $\underrightarrow{strong\_link\_to}$ \textbf{Type\_2\_Diabetes}}''. However, due to the attention bias mentioned above, the base model allocates low attention ($a'_2 \approx 8.0 \times 10^{-36}$) on the informative path, and thus fails to learn from such complex but useful evidences.

To reduce the negative effect of the attention biases and make full use of the UG path, we propose the following two training (or debiasing) strategies: Path Type Adaptive Pretraining (\S\ref{sec:pre}) and Complexity Ranking Guided Attention (\S\ref{sec:rank}). 



\subsection{Path Type Adaptive Pretraining}\label{sec:pre} 
As shown in Figure~\ref{fig:attpathtp}, the base model tends to bias toward KG paths. This indicates that the base model mainly relies on KG paths so that it is incapable of capturing informative features from Textual and Hybrid paths.
This bias will decrease the flexibility and adaptability of the base model to different types of paths.

To address this issue, we propose a debiasing strategy called Path Type Adaptive Pretraining.
In this strategy, we pretrain the base model sequentially using Textual, Hybrid, and KG Paths as path evidences, and then finetune it with all types of paths as illustrated in Figure~\ref{fig:pre_stra}. We hypothesize that this strategy can prevent the reliance on a single type of UG path and improve the capacity of extracting features from the entire UG paths, and thereby increase the performance.

\subsection{Complexity Ranking Guided Attention}\label{sec:rank} 
Similar to the bias towards KG paths, the base model also focuses on linguistically simple paths, as shown in Figure~\ref{fig:attlen}, even though complex ones are informative (e.g. $p_2$ in Figure~\ref{fig:ugexam}). We hypothesize that 
%restricting an attention span based on the path complexity can alleviate the negative effect of the attention bias. Specifically, 
restricting the attention span to the complex (simple) paths can force the model to pay attention to the complex (simple) paths, thereby effectively utilize them. Under this hypothesis, we propose a Complexity Ranking Guided Attention mechanism, as illustrated in Figure~\ref{fig:proposed_model}. 

Specifically, given a bag of paths $P_r=\{p_1,...,p_m\}$, we rank them according to their complexity scores ($\kappa $), which are calculated via $\kappa =\lambda_1\tau_1 + \lambda_2\tau_2 + ...$, where $\tau$ denotes the feature for capturing linguistic complexity (e.g., path length) and $\lambda$ is a corresponding weight, which is a hyperparameter. Sentence length (i.e., the number of tokens in a sentence) and lexical richness (i.e., the number of token types) are commonly used features for evaluating sentence complexity~\cite{brunato2018sentence}. Therefore, this work adopts them to calculate the complexity for a given path. 

Then, we group top $j$ most and least complex paths into a set of complex and simple paths respectively, where $j$ is a hyperparameter\footnote{In our experiments, we set $j$ as $30$ for NYT10 dataset and $50$ for Biomedical dataset.}. The set level representation is calculated via the Equation~\ref{eq:pathcomp}. 
%
\begin{equation}\label{eq:pathcomp}
\begin{split}
\mathbf{p}_{\mathrm{complex}\,or\,\mathrm{simple}}=\sum_{i\in top\_j \, or\, i\in last\_j}a'_i\mathbf{p}_i
\end{split}
\end{equation}
%
%
\begin{equation*}
\begin{split}
a'_i=\frac{\exp(\langle\mathbf{r}_{ht},\mathbf{x'}_i\rangle)}
{\sum_{k\in top\_j\,or\,k\in last\_j}\exp(\langle\mathbf{r}_{ht},\mathbf{x'}_k\rangle)},\\
\mathbf{x'}_i = \tanh(\mathbf{W}\mathbf{p}_i + \mathbf{b})
\end{split}
\end{equation*}
%
Finally, we concatenate the resulting representation $\mathbf{s}_{\mathrm{final}}$, $\mathbf{p}_{\mathrm{final}}$, $\mathbf{p}_{\mathrm{simple}}$ and $\mathbf{p}_{\mathrm{complex}}$ as the input to the relation classification layer. The conditional probability $P(e_1, r, e_2|S_r, P_r,\theta_S,\theta_P)$ is formulated via Equation~\ref{eq:pathall1}, where $\mathbf{o}=\mathbf{M}[\mathbf{s}_{\mathrm{all}}; \mathbf{p}_{\mathrm{all}};\mathbf{p}_{\mathrm{complex}};\mathbf{p}_{\mathrm{simple}}]+\mathbf{d}$. 
%
\begin{equation}\label{eq:pathall1}
P(e_1,r,e_2|S_r,P_r,\theta_S, \theta_P)=\frac{\exp([\mathbf{o}]_r)}
{\sum_{c=1}^{n_r}\exp([\mathbf{o}]_c)}
\end{equation}
%
%
%\begin{equation}\label{eq:pathall2}
%\mathbf{o}=\mathbf{M}[\mathbf{s}_{\mathrm{all}}; %\mathbf{p}_{\mathrm{all}};\mathbf{p}_{\mathrm{complex}};\mathbf{p}_{%\mathrm{simple}}]+\mathbf{d}
%\end{equation}
%

%
\begin{figure*}[t]
\centering
% \includegraphics[height=3.0cm, width=5.5cm]{graph/pre_stra.pdf}
\includegraphics[width=13.5cm]{graph/pretrain.pdf}
\caption{Path Type Adaptive Pretraining strategy, where ``Textual/Hybrid/KG P.'' represent Textual, Hybrid, and KG paths respectively, and ``Sent.'' represents the sentence evidences. In this strategy (\textbf{Pretrain}), instead of using all types of paths to train the base model in all iterations, we sequentially train the model with Textual, Hybrid and KG paths, and then finetune it with all types of paths.}
\label{fig:pre_stra}
\end{figure*}
%

\section{Experiments}

\subsection{Data}
\label{sec:data}
We evaluate our proposed framework on a biomedical dataset and NYT10 dataset~\cite{riedel2010modeling}. The statistics of both datasets is summarized in Table~\ref{tab:kb_sta}. We will detail both datasets as follows.

\textbf{Biomedical Dataset}. This datatset is created by linking biomedical KG with biomedical Text. We choose UMLS\footnote{\url{https://www.nlm.nih.gov/research/umls/}} and Medline corpus as the biomedical KG and Text respectively. UMLS is a frequently used biomedical knowledge base, while Medline corpus is a large collection of biomedical abstracts, both are developed and maintained by the U.S. National Library of Medicine\footnote{\url{https://www.nlm.nih.gov/}}. For identifying UMLS entity mentions in the Medline corpus, we use a state-of-the-art UMLS Named Entity Recognizer (NER), ScispaCy~\cite{neumann-etal-2019-scispacy}. The NER identifies UMLS concepts and annotates them by their corresponding UMLS Concept Unique Identifier (CUI) and entity types.

From the UMLS KG and the entity linked Medline corpus, we extract fact triplets (i.e., $(e_1, r, e_2)$) and corresponding sentence evidences containing $(e_1, e_2)$ under the restriction that: (1) each entity pair should be connected by a RO (RO stands for ``has Relationship Other than synonymous, narrower, or broader'') relationship; (2) each entity should belong to the following entity types: Protein, Gene, Disease or Syndrome, Enzyme, Chemical, Sign or Symptom and Pharmacologic Substance. Then we divide the collected triplets and sentence evidences into training and testing set according to the year when the source abstract of sentence evidence was published. The former is aligned to the years until 2008 and the latter to the years 2009 $\sim$ 2018, ensuring the testing set only contains the unobserved triplets. 

To simulate the noise in the real world, besides the ``related'' triplets, we also extract the ``unrelated'' triplets and sentence evidences based on a closed world assumption: pairs of entities not listed in a KG are regarded to have NA relation and sentences containing them are considered to be the NA sentence evidences. We divide the NA triplets and NA sentence evidences in the same way mentioned above. We use a subset of UMLS (see Appendix \S\ref{ap:sub})  and the Medline abstracts published until 2008 as the KG and Text respectively to create the UG for path retrieval. In addition, we use the same subset of UMLS triplets mentioned above to train the KG Encoder introduced in \S\ref{sec:evencoder}.

\textbf{NYT10}. The dataset is created by aligning Freebase relational facts with the New York Times Corpus. Sentence evidences from the year 2005 $\sim$ 2006 are used for training and the evidences from 2007 are used for testing. NYT10 dataset has been widely used by \cite{lin2016neural,ji2017distant,du2018multi,jat2018improving,du2018multi,han2018neural,han2018hierarchical,vashishth2018reside,ye2019distant,alt2019fine}. We use Freebase\footnote{From the entire Freebase, we only collect the triplets with the relations that are mentioned in NYT10 dataset for UG creation, ensuring not to overlap with testing set.} and ClueWeb12 with Freebase entity mention annotations~\cite{gabrilovich2013facc1} as the KG and Text to create the UG for path searching. In addition, following \cite{han2018neural}, we use FB60K for training the KG Encoder.
%
\begin{table}[t]
\centering
\scalebox{0.63}{
\begin{tabular}{c|c|c|c|c|c}
\hline
&\textbf{\#R} & \textbf{\#EP}& \textbf{\#Related EP} & \textbf{\#Sentence} &\textbf{\#UG Path} \\
\hline
\hline
Biomedical & 40 & \makecell{100,549 /\\ 21,081} & \makecell{10,936 /\\ 1,804} & \makecell{165,692 /\\ 28,912} & \makecell{12,854,696 /\\ 2,346,007}\\ \hline
NYT10 & 53 & \makecell{281,270 /\\ 96,678} & \makecell{18,252 /\\ 1,950} & \makecell{522,611 /\\  172,448} & \makecell{8,967,153 /\\  2,984,611} \\ \hline
\end{tabular}
}
\caption{Statistics of datasets in this work, where \textbf{R} and \textbf{EP} stand for the target Relation and Entity Pair, $\#_1$/$\#_2$ represent the number of training and testing data respectively.}
\label{tab:kb_sta}
\end{table}
%

\textbf{UG path search}. Given an entity pair ($e_1$, $e_2$), the UG path set $P_r$ is obtained by performing random walks over the UG from $e_1$ till $e_2$ with maximum step\footnote{We manually set the maximum step as 3.}.

\subsection{Settings}
\label{sec:settings}

We follow \cite{lin2016neural} and conduct the held-out evaluation, in which the model for DS-RE is evaluated by comparing the fact triplets identified from evidences (i.e., the bag of sentence evidences $S_r$ and the bag of UG path evidences $P_r$) with those in KG. Following the evaluation of previous works, we draw Precision-Recall curves and report the Area Under Curve (AUC) and Precision@N (P@N) metrics, which gives the percentage of correct triplets among top N ranked candidates.
The parameter settings of our experiments are detailed in Appendix \S\ref{ap:para}.

\begin{figure*}[t]
\centering
\begin{minipage}{.5\textwidth}
  \centering
  \includegraphics[width=\columnwidth]{graph/pr_med.pdf}
  \captionof{figure}{PR curves on Biomedical dataset.}
  \label{fig:prc_bio}
\end{minipage}%
\begin{minipage}{.5\textwidth}
  \centering
  \includegraphics[width=\columnwidth]{graph/pr_gen.pdf}
  \captionof{figure}{PR curves on NYT10 dataset.}
  \label{fig:prc_nyt}
\end{minipage}
\end{figure*}
%

\begin{table*}[t]
\centering
\scalebox{0.78}{
\begin{tabular}{lccccccccccccccc}
\hline
&\multicolumn{5}{c}{\textbf{Biomedical dataset}} &\multicolumn{7}{c}{\textbf{NYT10 dataset}}\\\cmidrule(lr){2-6}\cmidrule(lr){7-13}
\textbf{Model} & AUC & P@0.5k & P@1k & P@1.5k & P@2k & AUC & P@0.1k & P@0.2k & P@0.3k & P@0.5k & P@1k & P@2k\\\cmidrule(lr){1-6}\cmidrule(lr){7-13}\cmidrule(lr){1-6} \cmidrule(lr){7-13}
Sent & 9.6 & 30.0 & 20.8 & 18.7 & 16.3 & 36.6 & 81.0 & 73.5 & 68.3 & 62.0 & 53.8 & 40.2 \\\cmidrule(lr){1-6}\cmidrule(lr){7-13}
Sent+KG & 62.6 & 91.4 & 86.1 & 74.2 & 58.5 & 50.2 & 80.0 & 82.0 & 81.3 & 77.2 & 67.9 & 50.3\\\cmidrule(lr){1-6}\cmidrule(lr){7-13}
Sent+UG & 61.0 & 87.6 & 83.4 & 73.8 & 58.5 & 48.4 & 74.0 & 76.0 & 74.7 & 74.0 & 66.7 & 50.3\\\cmidrule(lr){1-6}\cmidrule(lr){7-13}
\makecell[l]{Sent+UG\\+Pretrain} & 70.1 & 95.4 & 89.7 & 76.3 & 60.4 & 52.7 & 83.0 & 82.0 & 80.3 & 78.6 & 70.4 & 52.6 \\\cmidrule(lr){1-6}\cmidrule(lr){7-13}
\makecell[l]{Sent+UG\\+Ranking} & 74.2 & 95.2 & 92.2 & 81.1 & 62.2 & 52.1 & 77.0 & 80.0 & 79.3 & 77.4 & 70.3 & 54.4 \\\cmidrule(lr){1-6}\cmidrule(lr){7-13}
\makecell[l]{Sent+UG\\+Ranking\\+Pretrain} & \textbf{77.5} & \textbf{95.4} & \textbf{93.1} & \textbf{83.9} & \textbf{64.4} & \textbf{55.0} & \textbf{86.0} & \textbf{84.0} & \textbf{83.3} & \textbf{80.4} & \textbf{71.9} & \textbf{54.5} \\\hline
\end{tabular}
}
\caption{P@N and AUC on Biomedical and NYT10 dataset (k=1000).}
\label{tab:patn_bio_nyt}
\end{table*}
%

%
\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{graph/pr_baselines.pdf}
\caption{PR curves of previous state-of-the-art methods and our proposed model on NYT10 dataset.}
\label{fig:prc_stnyt}
\end{figure}
%
\begin{table*}[t]
\centering
\scalebox{0.8}{
\begin{tabular}{c c c c c c c c}
\hline
\textbf{System} & AUC & P@0.1k & P@0.2k & P@0.3k & P@0.5k & P@1k & P@2k\\\hline\hline
Mintz\dag & 10.7 & 52.3 & 50.2 & 45.0 & 39.7 & 33.6 & 23.4 \\ \hline
PCNN+ATT\dag & 34.1 & 73.0 & 68.0 & 67.3 & 63.6 & 53.3 & 40.0 \\ \hline
RESIDE\dag & 41.5 & 81.8 & 75.4 & 74.3 & 69.7 & 59.3 & 45.0 \\ \hline
PCNN+HATT\ddag & 42.0 & 81.0 & 79.5 & 75.7 & 68.0 & 58.6 & 42.1 \\ \hline
DISTRE\dag & 42.2 & 68.0 & 67.0 & 65.3 & 65.0 & 60.2 & 47.9 \\ \hline
Sent+KG & 50.2 & 80.0 & 82.0 & 81.3 & 77.2 & 67.9 & 50.3 \\ \hline
Our Model & \textbf{55.0} & \textbf{86.0} & \textbf{84.0} & \textbf{83.3} & \textbf{80.4} & \textbf{71.9} & \textbf{54.5} \\ \hline
\end{tabular}
}
\caption{P@N and AUC from previous state-of-the-art DS-RE models and our proposed model on NYT10 dataset, where \dag represents that these results are quoted from \cite{alt2019fine} and \ddag indicates the results using the pretrained model from \cite{han2018hierarchical}.}
\label{tab:patn_stnyt}
\end{table*}
%

To demonstrate the effectiveness of our framework, we choose the model proposed by \citet{dai2019incorporating} as the baseline model, because this is the closest model in terms of incorporating multiple paths for DS-RE. Henceforth, ``Sent+KG'' is the baseline model, which uses both sentence evidences and KG paths. ``Sent+UG'' represents the base model in \S\ref{sec:evencoder} which takes UG paths instead of KG paths as path evidences. ``Sent+UG+Pretrain'' and ``Sent+UG+Ranking'' denote the base model trained with Path Type Adaptive Pretraining strategy and the base model with Complexity Ranking Guided Attention mechanism, respectively.
``Sent+UG+Ranking+Pretrain'' means the base model trained with both strategies.

\subsection{Results and Discussion}
\label{sec:results}

\textbf{Precision-Recall Curves}. The Precision-Recall (PR) curves of each model on the biomedical and NYT10 datasets are shown in Figure~\ref{fig:prc_bio} and Figure~\ref{fig:prc_nyt}, respectively.
The results show that: (1) ``Sent+UG'' does not have obvious advantages than ``Sent+KG'', illustrating that due to the biases discussed in \S\ref{sec:attbia}, simply applying UG paths on the base model has limited effect on improving the performance of DS-RE. (2) ``Sent+UG+Pretrain'' and ``Sent+UG+Ranking'' achieve better overall performance than ``Sent+KG'' on both datasets, especially when the recall is greater than $0.3$, demonstrating that UG has the potential to enhance the performance and the two proposed debiasing strategies are effective for exploiting the potential of UG for DS-RE. (3) ``Sent+UG+Ranking+Pretrain'' achieves the highest precision over the (almost) entire recall range on both datasets, proving that the two proposed strategies have a mutual complementary relationship on exploiting UG for DS-RE. This is understandable because the two proposed strategies deal with different types of biases, in addition, ``Pretrain'' helps the base model adapt to UG paths by effectively tuning its weights, while ``Ranking'' enhances the base model by adjusting its attention mechanism. (4) The consistent improvement on two datasets from different domains further proves the validity of our proposed methods. 

\textbf{AUC and P@N Evaluation}. Table~\ref{tab:patn_bio_nyt} further presents the results in terms of AUC and P@N. From them, we have similar observation to the PR curves.
%: ``Sent+UG'' is inferior to ``Sent+KG''; but ``Sent+UG+Ranking+Pretrain'' achieves the best performance. 
We also observe that the effectiveness of UG paths is more pronounced on Biomedical dataset than on NYT10 dataset. We speculate that compared to the generic NYT10 dataset, further Background Knowledge (BK) is needed to identify relations from Biomedical dataset, and UG paths could be utilized as the BK to facilitate the scientific DS-RE.

\textbf{Comparison with State-of-the-art Baselines on NYT10}. To demonstrate the effectiveness of our proposed model, we also compare it against the following baselines on NYT10 dataset: Mintz~\cite{mintz2009distant}, MultiR~\cite{hoffmann2011knowledge}, MIMLRE~\cite{surdeanu2012multi}, PCNN~\cite{zeng2015distant}, PCNN+ATT~\cite{lin2016neural}, BGWA~\cite{jat2018improving}, PCNN+HATT~\cite{han2018hierarchical}, RESIDE~\cite{vashishth2018reside}, DISTRE~\cite{alt2019fine} and Sent+KG~\cite{dai2019incorporating}.
The results shown in Figure~\ref{fig:prc_stnyt} and Table~\ref{tab:patn_stnyt} indicate that: (1) our selected base model, ``Sent+KG'', is a strong baseline because it significantly outperforms other state-of-the-art models; and (2) our model can effectively take advantage of the rich UG paths for DS-RE because it beats the strong baseline and achieves a new state-of-the-art result on the commonly used DS-RE dataset.

%
\begin{table}[ht]
\centering
\resizebox{\columnwidth}{3.0cm}{
\begin{tabular}{cccl}
\hline
Base & Prop. & \textbf{Biomedical Triplet} \\ \hline
\xmark & \cmark & ( \colorbox{green!30}{Beta-2...Gene}, gene\_associated\_with\_disease, \colorbox{red!30}{Asthma})\\ \hline
\multicolumn{2}{c}{} & \textbf{Multi-hop Path} \\ \hline
Low & \textbf{High} & \makecell[l]{$hop_1$: \it ``The human \colorbox{green!30}{Beta-2...Gene} is responsible for \\ \it the binding of endogenous \colorbox{black!10}{Catecholamine} and their ...'' \\ $hop_2$: \it ``\colorbox{black!10}{Catecholamine} chemical structure of \colorbox{orange!30}{Epinephrine}'' \\ $hop_3$: \it `` \colorbox{orange!30}{Epinephrine} may treat \colorbox{red!30}{Asthma}''.} \\  \hline\hline
Base & Prop. & \textbf{NYT10 Triplet} \\ \hline
\xmark & \cmark & ( \colorbox{green!30}{San\_Francisco}, /location/contains, \colorbox{red!30}{Noe\_Valley})\\ \hline
\multicolumn{2}{c}{} & \textbf{Multi-hop Path} \\ \hline
Low & \textbf{High} & \makecell[l]{$hop_1$: \it ``\colorbox{green!30}{San\_Francisco} /location/contains \colorbox{black!10}{Fort\_Point}'' \\ $hop_2$: \it ``Surf spots and surfing regions include Northern CA, \\ \it the \colorbox{orange!30}{Bay\_Area}, San Francisco, Ocean Beach and \colorbox{black!10}{Fort Point}'' \\ $hop_3$: \it ``\colorbox{orange!30}{Bay\_Area} /location/contains \colorbox{red!30}{Noe\_Valley}''} \\  \hline
\end{tabular}
}
\caption{\label{tab:casestudy} Some examples of attention distribution over paths from ``Sent+UG'' (Base) and ``Sent+UG+Ranking+Pretrain'' (Prop.), where \cmark (or \xmark) represents the correct (or incorrect) prediction of the target relation.}
\end{table}
%

\textbf{Case Study}. Table~\ref{tab:casestudy} shows the UG path examples that are scored with highest (``High'') or lowest (or lower than $1.0\times10^{-3}$) (``Low'') attention by the base model and our proposed framework. The paths in the table generally mean ``\colorbox{green!30}{\it Beta-2... Gene} $\underrightarrow{is\_responsible\_for}$ \colorbox{black!10}{\it Catecholamine} $\underrightarrow{is\_the\_chemical\_class\_of}$ \colorbox{orange!30}{\it Epinephrine} $\underrightarrow{may\_treat}$ \colorbox{red!30}{\it Asthma}'' and ``\colorbox{green!30}{\it San\_Francisco} $\underrightarrow{contains}$ \colorbox{black!10}{\it Fort\_Point} $\underrightarrow{equal\_status}$ \colorbox{orange!30}{\it Bay\_Area} $\underrightarrow{contains}$ \colorbox{red!30}{\it Noe\_Valley}'', and thus can be seen as the useful path evidences for identifying \emph{gene\_associated\_with\_disease} and \emph{/location/contains} relation respectively. These examples indicate that our proposed training strategies could help the base model attend such informative UG paths so that it can correctly identify the target relation.

\section{Conclusion and Future Work}
We have introduced UG paths as extra evidences for the task of DS-RE from text. In order to fully take advantage of the rich UG paths, we have proposed two training (or debiasing) strategies: Path Type Adaptive Pretraining and Complexity Ranking Guided Attention mechanism. We have conducted experiments on both biomedical and NYT10 datasets. The results show that the two proposed methods are effective for exploiting the potential of UG paths for improving the performance of DS-RE. 
%and improve the performance consistently and significantly. 

In the future, we plan to carry out the following steps: (1) we further investigate how the proposed training methods influence the performance via manual analysis so as to better the efficiency; and (2) instead of random walk, we may collect UG paths by adopting more sophisticated mechanisms such as training a path searching agent via reinforcement learning to prevent redundant and noisy paths.  

\section*{Acknowledgement}
This work was supported by JST CREST Grant Number JPMJCR1513, Japan and KAKENHI Grant Number 16H06614. We would like to thank Benjamin Heinzerling, Shota Sasaki and other collaborators for their useful comments and suggestions.

% include your own bib file like this:
\bibliography{eacl_2021}
\bibliographystyle{acl_natbib}

\newpage
\appendix
\section{Appendix}
\subsection{CNN-Max}\label{ap:cnnmax}
Convolutional Neural Network with Max pooling layer (CNN-Max) is adopted to derive the sentence representation $\mathbf{s}$ and path representation $\mathbf{p}$. Specifically, vector representation $\mathbf{v}_t$ for each word $w_t$ is calculated via Equation~\ref{eq:cnnemb1}, where $\mathbf{W}_{emb}^w$ is a word embedding projection matrix~\cite{mikolov2013distributed}, $\mathbf{W}_{emb}^{wp}$ is a word position embedding projection matrix~\cite{zeng2014relation}, $\mathbf{x}_t^w$ is a one-hot word representation and $\mathbf{x}_t^{wp}$ is a one-hot word position representation, which indicates the relative distance between the current word and the target entity pair. 
%
\begin{equation}\label{eq:cnnemb1}
\mathbf{v}_t=[\mathbf{v}_t^w; \mathbf{v}_t^{wp1}; \mathbf{v}_t^{wp2}],
\end{equation}
%
%
\begin{equation*}
\begin{split}
\mathbf{v}_t^w=\mathbf{W}_{emb}^w\mathbf{x}_t^w, \\
\mathbf{v}_t^{wp1}=\mathbf{W}_{emb}^{wp}\mathbf{x}_t^{wp1}, \\
\mathbf{v}_t^{wp2}=\mathbf{W}_{emb}^{wp}\mathbf{x}_t^{wp2}
\end{split}
\end{equation*}
%
The sentence representation $\mathbf{s}$ and path representation $\mathbf{p}$ are formulated via the Equation~\ref{eq:cnnmax}, where $\mathbf{W}^{sent}$ (or $\mathbf{W}^{path}$) is the convolution kernal, $\mathbf{b}^{sent}$ (or $\mathbf{b}^{path}$) is the corresponding bias vector, $\mathbf{v}^{sent}_t$ (or $\mathbf{v}^{path}_t$) is the vector for each word $w_t$ in a sentence (or path), $[vec]_i$ is the $i$-th value of $vec$, $\nu$ is the dimensionality of $\mathbf{s}$ and $\mathbf{p}$, and $k$ is the convolutional window size.
%
\begin{equation}\label{eq:cnnmax}
\begin{split}
[\mathbf{s}]_i = \max\limits_{t}\{[\mathbf{h}^{sent}_{t}]_{i}\},\ \forall i=1, ..., \nu \\
[\mathbf{p}]_i = \max\limits_{t}\{[\mathbf{h}^{path}_{t}]_{i}\},\ \forall i=1, ..., \nu \\
\end{split}
\end{equation}
\begin{equation*}
\begin{split}
\mathbf{h}^{sent}_t=\tanh(\mathbf{W}^{sent}\mathbf{z}^{sent}_t + \mathbf{b}^{sent}),\\
\mathbf{h}^{path}_t=\tanh(\mathbf{W}^{path}\mathbf{z}^{path}_t + \mathbf{b}^{path}),\\
\mathbf{z}^{sent}_t=[\mathbf{v}^{sent}_{t-(k-1)/2}; ... ; \mathbf{v}^{sent}_{t+(k-1)/2}],\\
\mathbf{z}^{path}_t=[\mathbf{v}^{path}_{t-(k-1)/2}; ... ; \mathbf{v}^{path}_{t+(k-1)/2}]
\end{split}
\end{equation*}
%

\subsection{Parameter Settings}\label{ap:para}
All of the hyperparameters used in our experiments are listed in Table~\ref{tab:hypara}. Most of them follow the hyperparameter setting in \cite{dai2019incorporating} and \cite{han2018neural}. We use a Word2Vec model\footnote{Gensim word2vec implementation: \url{https://radimrehurek.com/gensim/models/word2vec.html}} to train the word embeddings on the UMLS entity linked corpus for the biomedical dataset, and adopt the word embeddings released by \cite{lin2016neural} for NYT10 dataset. We apply Stochastic Gradient Descent (SGD) to optimize the proposed DS-RE model.
%
\begin{table}[t]
\centering
\scalebox{0.8}{
\begin{tabular}{|l|c|c|}
\hline
\textbf{Hyperparameter} & \textbf{Biomedical} & \textbf{NYT10} \\\hline
word embedding dimension & 50 & 50 \\ \hline
KG embedding dimension & 50 & 50 \\ \hline
position embedding dimension & 5 & 5 \\ \hline
CNN window size & 3 & 3 \\ \hline
CNN filter number & 100 & 230 \\ \hline
dropout rate & 0.5 & 0.5 \\ \hline
\makecell[l]{learning rate \\(for sentences and paths)} & 0.02 & 0.05 \\ \hline
learning rate (for KG) & 0.05 & 0.001 \\ \hline
batch size & 50 & 160 \\ \hline
\end{tabular}
}
\caption{Hyperparameters used in our experiments.}
\label{tab:hypara}
\end{table}
%

\subsection{Subset of UMLS}\label{ap:sub}
Besides the 7 entity types mentioned above, we also use other 22 entity types, as listed in Table~\ref{tab:umls_tp}, to collect the UMLS triplets that are connected by RO relationship, ensuring all testing triplets are removed. The main reasons to manually restrict the entity type is because (1) we observe that most of the Medline abstracts discuss the relationship among these entity types; (2) these concrete entities could prevent semantic drift while searching UG paths.

%
\begin{table}[t]
\centering
\scalebox{1.0}{
\begin{tabular}{c}
\hline
\textbf{Selected Entity Types} \\\hline
\makecell[l]{Antibiotic, Biologically Active Substance, \\ Bacterium, Organ, Cell Component,\\ Cell Function, Cell, Clinical Drug, Ion, \\ Eukaryote, Food, Genetic Function \\ Hazardous or Poisonous Substance, \\ Hormone, Immunologic Factor,\\ Inorganic Chemical, Organic Chemical, \\ Pathologic Function, Receptor, \\ Steroid, Virus and Vitamin.} \\ \hline
\end{tabular}
}
\caption{List of selected UMLS entity types.}
\label{tab:umls_tp}
\end{table}
%

\end{document}

eacl_2021.bbl
\begin{thebibliography}{31}
\expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi

\bibitem[{Alt et~al.(2019)Alt, H{\"u}bner, and Hennig}]{alt2019fine}
Christoph Alt, Marc H{\"u}bner, and Leonhard Hennig. 2019.
\newblock Fine-tuning pre-trained transformer language models to distantly
  supervised relation extraction.
\newblock \emph{arXiv preprint arXiv:1906.08646}.

\bibitem[{Bordes et~al.(2013)Bordes, Usunier, Garcia-Duran, Weston, and
  Yakhnenko}]{bordes2013translating}
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana
  Yakhnenko. 2013.
\newblock Translating embeddings for modeling multi-relational data.
\newblock In \emph{Advances in neural information processing systems}, pages
  2787--2795.

\bibitem[{Brunato et~al.(2018)Brunato, De~Mattei, Dell’Orletta, Iavarone, and
  Venturi}]{brunato2018sentence}
Dominique Brunato, Lorenzo De~Mattei, Felice Dell’Orletta, Benedetta
  Iavarone, and Giulia Venturi. 2018.
\newblock Is this sentence difficult? do you agree?
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
  Natural Language Processing}, pages 2690--2699.

\bibitem[{Dai et~al.(2019)Dai, Inoue, Reisert, Ryo, and
  Inui}]{dai2019incorporating}
Qin Dai, Naoya Inoue, Paul Reisert, Takahashi Ryo, and Kentaro Inui. 2019.
\newblock \href {http://jaslli.org/files/proceedings/03_paclic33_postconf.pdf}
  {Incorporating chains of reasoning over knowledge graph for distantly
  supervised biomedical knowledge acquisition}.
\newblock In \emph{Proceedings of the 33nd Pacific Asia Conference on Language,
  Information and Computation (PACLIC33)}, pages 19--28, Hakodate, Japan.
  Waseda Institute for the Study of Language and Information.

\bibitem[{Das et~al.(2017{\natexlab{a}})Das, Neelakantan, Belanger, and
  McCallum}]{das2017chains}
Rajarshi Das, Arvind Neelakantan, David Belanger, and Andrew McCallum.
  2017{\natexlab{a}}.
\newblock Chains of reasoning over entities, relations, and text using
  recurrent neural networks.
\newblock In \emph{Proceedings of the 15th Conference of the European Chapter
  of the Association for Computational Linguistics: Volume 1, Long Papers},
  pages 132--141.

\bibitem[{Das et~al.(2017{\natexlab{b}})Das, Zaheer, Reddy, and
  McCallum}]{das2017question}
Rajarshi Das, Manzil Zaheer, Siva Reddy, and Andrew McCallum.
  2017{\natexlab{b}}.
\newblock Question answering on knowledge bases and text using universal schema
  and memory networks.
\newblock In \emph{Proceedings of the 55th Annual Meeting of the Association
  for Computational Linguistics (Volume 2: Short Papers)}, pages 358--365.

\bibitem[{Du et~al.(2018)Du, Han, Way, and Wan}]{du2018multi}
Jinhua Du, Jingguang Han, Andy Way, and Dadong Wan. 2018.
\newblock Multi-level structured self-attentions for distantly supervised
  relation extraction.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
  Natural Language Processing}, pages 2216--2225.

\bibitem[{Gabrilovich et~al.(2013)Gabrilovich, Ringgaard, and
  Subramanya}]{gabrilovich2013facc1}
Evgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya. 2013.
\newblock Facc1: Freebase annotation of clueweb corpora, version 1 (release
  date 2013-06-26, format version 1, correction level 0).

\bibitem[{Han et~al.(2018{\natexlab{a}})Han, Liu, and Sun}]{han2018neural}
Xu~Han, Zhiyuan Liu, and Maosong Sun. 2018{\natexlab{a}}.
\newblock Neural knowledge acquisition via mutual attention between knowledge
  graph and text.
\newblock In \emph{Thirty-Second AAAI Conference on Artificial Intelligence}.

\bibitem[{Han et~al.(2018{\natexlab{b}})Han, Yu, Liu, Sun, and
  Li}]{han2018hierarchical}
Xu~Han, Pengfei Yu, Zhiyuan Liu, Maosong Sun, and Peng Li. 2018{\natexlab{b}}.
\newblock Hierarchical relation extraction with coarse-to-fine grained
  attention.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
  Natural Language Processing}, pages 2236--2245.

\bibitem[{Hoffmann et~al.(2011)Hoffmann, Zhang, Ling, Zettlemoyer, and
  Weld}]{hoffmann2011knowledge}
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel~S Weld.
  2011.
\newblock Knowledge-based weak supervision for information extraction of
  overlapping relations.
\newblock In \emph{Proceedings of the 49th Annual Meeting of the Association
  for Computational Linguistics: Human Language Technologies-Volume 1}, pages
  541--550. Association for Computational Linguistics.

\bibitem[{Jat et~al.(2018)Jat, Khandelwal, and Talukdar}]{jat2018improving}
Sharmistha Jat, Siddhesh Khandelwal, and Partha Talukdar. 2018.
\newblock Improving distantly supervised relation extraction using word and
  entity based attention.
\newblock \emph{arXiv preprint arXiv:1804.06987}.

\bibitem[{Ji et~al.(2017)Ji, Liu, He, and Zhao}]{ji2017distant}
Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. 2017.
\newblock Distant supervision for relation extraction with sentence-level
  attention and entity descriptions.
\newblock In \emph{Thirty-First AAAI Conference on Artificial Intelligence}.

\bibitem[{Jia et~al.(2019)Jia, Dai, Xiao, and Wu}]{jia2019arnor}
Wei Jia, Dai Dai, Xinyan Xiao, and Hua Wu. 2019.
\newblock Arnor: attention regularization based noise reduction for distant
  supervision relation classification.
\newblock In \emph{Proceedings of the 57th Annual Meeting of the Association
  for Computational Linguistics}, pages 1399--1408.

\bibitem[{Lin et~al.(2017)Lin, Liu, and Sun}]{lin2017neural}
Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2017.
\newblock Neural relation extraction with multi-lingual attention.
\newblock In \emph{Proceedings of the 55th Annual Meeting of the Association
  for Computational Linguistics (Volume 1: Long Papers)}, pages 34--43.

\bibitem[{Lin et~al.(2016)Lin, Shen, Liu, Luan, and Sun}]{lin2016neural}
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016.
\newblock Neural relation extraction with selective attention over instances.
\newblock In \emph{Proceedings of the 54th Annual Meeting of the Association
  for Computational Linguistics (Volume 1: Long Papers)}, volume~1, pages
  2124--2133.

\bibitem[{Mikolov et~al.(2013)Mikolov, Sutskever, Chen, Corrado, and
  Dean}]{mikolov2013distributed}
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg~S Corrado, and Jeff Dean. 2013.
\newblock Distributed representations of words and phrases and their
  compositionality.
\newblock In \emph{Advances in neural information processing systems}, pages
  3111--3119.

\bibitem[{Min et~al.(2013)Min, Grishman, Wan, Wang, and
  Gondek}]{min2013distant}
Bonan Min, Ralph Grishman, Li~Wan, Chang Wang, and David Gondek. 2013.
\newblock Distant supervision for relation extraction with an incomplete
  knowledge base.
\newblock In \emph{Proceedings of the 2013 Conference of the North American
  Chapter of the Association for Computational Linguistics: Human Language
  Technologies}, pages 777--782.

\bibitem[{Mintz et~al.(2009)Mintz, Bills, Snow, and
  Jurafsky}]{mintz2009distant}
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009.
\newblock Distant supervision for relation extraction without labeled data.
\newblock In \emph{Proceedings of the Joint Conference of the 47th Annual
  Meeting of the ACL and the 4th International Joint Conference on Natural
  Language Processing of the AFNLP: Volume 2-Volume 2}, pages 1003--1011.
  Association for Computational Linguistics.

\bibitem[{Neelakantan et~al.(2015)Neelakantan, Roth, and
  McCallum}]{neelakantan-etal-2015-compositional}
Arvind Neelakantan, Benjamin Roth, and Andrew McCallum. 2015.
\newblock \href {https://doi.org/10.3115/v1/P15-1016} {Compositional vector
  space models for knowledge base completion}.
\newblock In \emph{Proceedings of the 53rd Annual Meeting of the Association
  for Computational Linguistics and the 7th International Joint Conference on
  Natural Language Processing (Volume 1: Long Papers)}, pages 156--166,
  Beijing, China. Association for Computational Linguistics.

\bibitem[{Neumann et~al.(2019)Neumann, King, Beltagy, and
  Ammar}]{neumann-etal-2019-scispacy}
Mark Neumann, Daniel King, Iz~Beltagy, and Waleed Ammar. 2019.
\newblock \href {https://doi.org/10.18653/v1/W19-5034} {{S}cispa{C}y: {F}ast
  and {R}obust {M}odels for {B}iomedical {N}atural {L}anguage {P}rocessing}.
\newblock In \emph{Proceedings of the 18th BioNLP Workshop and Shared Task},
  pages 319--327, Florence, Italy. Association for Computational Linguistics.

\bibitem[{Quirk and Poon(2017)}]{quirk2017distant}
Chris Quirk and Hoifung Poon. 2017.
\newblock Distant supervision for relation extraction beyond the sentence
  boundary.
\newblock In \emph{Proceedings of the 15th Conference of the European Chapter
  of the Association for Computational Linguistics: Volume 1, Long Papers},
  pages 1171--1182.

\bibitem[{Riedel et~al.(2010)Riedel, Yao, and McCallum}]{riedel2010modeling}
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010.
\newblock Modeling relations and their mentions without labeled text.
\newblock In \emph{Joint European Conference on Machine Learning and Knowledge
  Discovery in Databases}, pages 148--163. Springer.

\bibitem[{Sun et~al.(2019)}]{sun2019leveraging}
Huan Sun et~al. 2019.
\newblock Leveraging 2-hop distant supervision from table entity pairs for
  relation extraction.
\newblock In \emph{Proceedings of the 2019 Conference on Empirical Methods in
  Natural Language Processing and the 9th International Joint Conference on
  Natural Language Processing (EMNLP-IJCNLP)}, pages 410--420.

\bibitem[{Surdeanu et~al.(2012)Surdeanu, Tibshirani, Nallapati, and
  Manning}]{surdeanu2012multi}
Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher~D Manning.
  2012.
\newblock Multi-instance multi-label learning for relation extraction.
\newblock In \emph{Proceedings of the 2012 joint conference on empirical
  methods in natural language processing and computational natural language
  learning}, pages 455--465. Association for Computational Linguistics.

\bibitem[{Toutanova et~al.(2015)Toutanova, Chen, Pantel, Poon, Choudhury, and
  Gamon}]{toutanova2015representing}
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi
  Choudhury, and Michael Gamon. 2015.
\newblock Representing text for joint embedding of text and knowledge bases.
\newblock In \emph{Proceedings of the 2015 conference on empirical methods in
  natural language processing}, pages 1499--1509.

\bibitem[{Vashishth et~al.(2018)Vashishth, Joshi, Prayaga, Bhattacharyya, and
  Talukdar}]{vashishth2018reside}
Shikhar Vashishth, Rishabh Joshi, Sai~Suman Prayaga, Chiranjib Bhattacharyya,
  and Partha Talukdar. 2018.
\newblock Reside: Improving distantly-supervised neural relation extraction
  using side information.
\newblock In \emph{Proceedings of the 2018 Conference on Empirical Methods in
  Natural Language Processing}, pages 1257--1266.

\bibitem[{Ye and Ling(2019)}]{ye2019distant}
Zhi-Xiu Ye and Zhen-Hua Ling. 2019.
\newblock Distant supervision relation extraction with intra-bag and inter-bag
  attentions.
\newblock In \emph{Proceedings of the 2019 Conference of the North American
  Chapter of the Association for Computational Linguistics: Human Language
  Technologies, Volume 1 (Long and Short Papers)}, pages 2810--2819.

\bibitem[{Zeng et~al.(2015)Zeng, Liu, Chen, and Zhao}]{zeng2015distant}
Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015.
\newblock Distant supervision for relation extraction via piecewise
  convolutional neural networks.
\newblock In \emph{Proceedings of the 2015 Conference on Empirical Methods in
  Natural Language Processing}, pages 1753--1762.

\bibitem[{Zeng et~al.(2014)Zeng, Liu, Lai, Zhou, Zhao
  et~al.}]{zeng2014relation}
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao, et~al. 2014.
\newblock Relation classification via convolutional deep neural network.
\newblock In \emph{COLING}, pages 2335--2344.

\bibitem[{Zeng et~al.(2017)Zeng, Lin, Liu, and Sun}]{zeng2017incorporating}
Wenyuan Zeng, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2017.
\newblock Incorporating relation paths in neural relation extraction.
\newblock In \emph{Proceedings of the 2017 Conference on Empirical Methods in
  Natural Language Processing}, pages 1768--1777.

\end{thebibliography}

eacl_2021.bib

%% This BibTeX bibliography file was created using BibDesk.
%% http://bibdesk.sourceforge.net/

%% Created for Daiqin at 2018-02-23 17:45:32 +0900 


%% Saved with string encoding Unicode (UTF-8) 



@inproceedings{bird2008acl,
	Author = {Bird, Steven and Dale, Robert and Dorr, Bonnie J and Gibson, Bryan R and Joseph, Mark Thomas and Kan, Min-Yen and Lee, Dongwon and Powley, Brett and Radev, Dragomir R and Tan, Yee Fan and others},
	Booktitle = {LREC},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Title = {The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics.},
	Year = {2008}}

@inproceedings{mikolov2013distributed,
	Author = {Mikolov, Tomas and Sutskever, Ilya and Chen, Kai and Corrado, Greg S and Dean, Jeff},
	Booktitle = {Advances in neural information processing systems},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Pages = {3111--3119},
	Title = {Distributed representations of words and phrases and their compositionality},
	Year = {2013}}

@book{fellbaum98_wn,
	Author = {Fellbaum, Christian},
	Booktitle = {MIT Press},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Mendeley-Groups = {My references},
	Publisher = {MIT Press},
	Title = {{WordNet: An Electronic Lexical Database}},
	Year = {1998}}

@article{kumar2017survey,
	Author = {Kumar, Shantanu},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {arXiv preprint arXiv:1705.03645},
	Title = {A Survey of Deep Learning Methods for Relation Extraction},
	Year = {2017}}

@article{zhou2014biomedical,
	Author = {Zhou, Deyu and Zhong, Dayou and He, Yulan},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {Computational and mathematical methods in medicine},
	Publisher = {Hindawi Publishing Corporation},
	Title = {Biomedical relation extraction: from binary to complex},
	Volume = {2014},
	Year = {2014}}

@phdthesis{teufel2000argumentative,
	Author = {Teufel, Simone and others},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	School = {University of Edinburgh},
	Title = {Argumentative zoning: Information extraction from scientific text},
	Year = {2000}}

@article{zhang2015relation,
	Author = {Zhang, Dongxu and Wang, Dong},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {arXiv preprint arXiv:1508.01006},
	Title = {Relation classification via recurrent neural network},
	Year = {2015}}

@article{santos2015classifying,
	Author = {Santos, Cicero Nogueira dos and Xiang, Bing and Zhou, Bowen},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {arXiv preprint arXiv:1504.06580},
	Title = {Classifying relations by ranking with convolutional neural networks},
	Year = {2015}}

@article{gu2017chemical,
	Author = {Gu, Jinghang and Sun, Fuqing and Qian, Longhua and Zhou, Guodong},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {Database},
	Publisher = {Oxford University Press},
	Title = {Chemical-induced disease relation extraction via convolutional neural network},
	Volume = {2017},
	Year = {2017}}

@inproceedings{ammar2017ai2,
	Author = {Ammar, Waleed and Peters, Matthew and Bhagavatula, Chandra and Power, Russell},
	Booktitle = {Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Pages = {592--596},
	Title = {The AI2 system at SemEval-2017 Task 10 (ScienceIE): semi-supervised end-to-end entity and relation extraction},
	Year = {2017}}

@inproceedings{boschee2005automatic,
	Author = {Boschee, Elizabeth and Weischedel, Ralph and Zamanian, Alex},
	Booktitle = {Proceedings of the International Conference on Intelligence Analysis},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Organization = {Citeseer},
	Title = {Automatic information extraction},
	Volume = {71},
	Year = {2005}}

@inproceedings{suchanek2006combining,
	Author = {Suchanek, Fabian M and Ifrim, Georgiana and Weikum, Gerhard},
	Booktitle = {Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Organization = {ACM},
	Pages = {712--717},
	Title = {Combining linguistic and statistical analysis to extract relations from web documents},
	Year = {2006}}

@inproceedings{guodong2005exploring,
	Author = {GuoDong, Zhou and Jian, Su and Jie, Zhang and Min, Zhang},
	Booktitle = {Proceedings of the 43rd annual meeting on association for computational linguistics},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Organization = {Association for Computational Linguistics},
	Pages = {427--434},
	Title = {Exploring various knowledge in relation extraction},
	Year = {2005}}

@inproceedings{chan2010exploiting,
	Author = {Chan, Yee Seng and Roth, Dan},
	Booktitle = {Proceedings of the 23rd International Conference on Computational Linguistics},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Organization = {Association for Computational Linguistics},
	Pages = {152--160},
	Title = {Exploiting background knowledge for relation extraction},
	Year = {2010}}

@inproceedings{zeng2014relation,
	Author = {Zeng, Daojian and Liu, Kang and Lai, Siwei and Zhou, Guangyou and Zhao, Jun and others},
	Booktitle = {COLING},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Pages = {2335--2344},
	Title = {Relation Classification via Convolutional Deep Neural Network.},
	Year = {2014}}

@article{miwa2016end,
	Author = {Miwa, Makoto and Bansal, Mohit},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {arXiv preprint arXiv:1601.00770},
	Title = {End-to-end relation extraction using lstms on sequences and tree structures},
	Year = {2016}}

@article{hahn2016before,
	Author = {Hahn-Powell, Gus and Bell, Dane and Valenzuela-Esc{\'a}rcega, Marco A and Surdeanu, Mihai},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {arXiv preprint arXiv:1606.08089},
	Title = {This before That: Causal Precedence in the Biomedical Domain},
	Year = {2016}}

@article{arora2016simple,
	Author = {Arora, Sanjeev and Liang, Yingyu and Ma, Tengyu},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Title = {A simple but tough-to-beat baseline for sentence embeddings},
	Year = {2016}}

@inproceedings{tateisi2014annotation,
	Author = {Tateisi, Yuka and Shidahara, Yo and Miyao, Yusuke and Aizawa, Akiko},
	Booktitle = {LREC},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Pages = {1423--1429},
	Title = {Annotation of Computer Science Papers for Semantic Relation Extraction.},
	Year = {2014}}

@article{augenstein2017semeval,
	Author = {Augenstein, Isabelle and Das, Mrinal and Riedel, Sebastian and Vikraman, Lakshmi and McCallum, Andrew},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {arXiv preprint arXiv:1704.02853},
	Title = {SemEval 2017 Task 10: ScienceIE-Extracting Keyphrases and Relations from Scientific Publications},
	Year = {2017}}

@article{morris1991lexical,
	Author = {Morris, Jane and Hirst, Graeme},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {Computational linguistics},
	Number = {1},
	Pages = {21--48},
	Publisher = {MIT Press},
	Title = {Lexical cohesion computed by thesaural relations as an indicator of the structure of text},
	Volume = {17},
	Year = {1991}}

@article{cohen2017bionlp,
	Author = {Cohen, Kevin Bretonnel and Demner-Fushman, Dina and Ananiadou, Sophia and Tsujii, Junichi},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {BioNLP 2017},
	Title = {BioNLP 2017},
	Year = {2017}}

@inproceedings{mascarell2017lexical,
	Author = {Mascarell, Laura},
	Booktitle = {Proceedings of the Third Workshop on Discourse in Machine Translation},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Pages = {99--109},
	Title = {Lexical Chains meet Word Embeddings in Document-level Statistical Machine Translation},
	Year = {2017}}

@inproceedings{bergstra2010theano,
	Author = {Bergstra, James and Breuleux, Olivier and Bastien, Fr{\'e}d{\'e}ric and Lamblin, Pascal and Pascanu, Razvan and Desjardins, Guillaume and Turian, Joseph and Warde-Farley, David and Bengio, Yoshua},
	Booktitle = {Proc. 9th Python in Science Conf},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Pages = {1--7},
	Title = {Theano: A CPU and GPU math compiler in Python},
	Year = {2010}}

@inproceedings{chen2006relation,
	Author = {Chen, Jinxiu and Ji, Donghong and Tan, Chew Lim and Niu, Zhengyu},
	Booktitle = {Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Organization = {Association for Computational Linguistics},
	Pages = {129--136},
	Title = {Relation extraction using label propagation based semi-supervised learning},
	Year = {2006}}

@inproceedings{nguyen2015relation,
	Author = {Nguyen, Thien Huu and Grishman, Ralph},
	Booktitle = {VS@ HLT-NAACL},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Pages = {39--48},
	Title = {Relation Extraction: Perspective from Convolutional Neural Networks.},
	Year = {2015}}

@article{kim2008corpus,
	Author = {Kim, Jin-Dong and Ohta, Tomoko and Tsujii, Jun'ichi},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {BMC bioinformatics},
	Number = {1},
	Pages = {10},
	Publisher = {BioMed Central},
	Title = {Corpus annotation for mining biomedical events from literature},
	Volume = {9},
	Year = {2008}}

@inproceedings{angeli2014combining,
	Author = {Angeli, Gabor and Tibshirani, Julie and Wu, Jean and Manning, Christopher D},
	Booktitle = {EMNLP},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Pages = {1556--1567},
	Title = {Combining Distant and Partial Supervision for Relation Extraction.},
	Year = {2014}}

@inproceedings{wang2006automatic,
	Author = {Wang, Ting and Li, Yaoyong and Bontcheva, Kalina and Cunningham, Hamish and Wang, Ji},
	Booktitle = {European Semantic Web Conference},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Organization = {Springer},
	Pages = {215--229},
	Title = {Automatic extraction of hierarchical relations from text},
	Year = {2006}}

@inproceedings{gupta2016table,
	Author = {Gupta, Pankaj and Sch{\"u}tze, Hinrich and Andrassy, Bernt},
	Booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Pages = {2537--2547},
	Title = {Table filling multi-task recurrent neural network for joint entity and relation extraction},
	Year = {2016}}

@article{yin2017comparative,
	Author = {Yin, Wenpeng and Kann, Katharina and Yu, Mo and Sch{\"u}tze, Hinrich},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {arXiv preprint arXiv:1702.01923},
	Title = {Comparative study of cnn and rnn for natural language processing},
	Year = {2017}}

@article{elman1990finding,
	Author = {Elman, Jeffrey L},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {Cognitive science},
	Number = {2},
	Pages = {179--211},
	Publisher = {Wiley Online Library},
	Title = {Finding structure in time},
	Volume = {14},
	Year = {1990}}

@article{lecun1998gradient,
	Author = {LeCun, Yann and Bottou, L{\'e}on and Bengio, Yoshua and Haffner, Patrick},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {Proceedings of the IEEE},
	Number = {11},
	Pages = {2278--2324},
	Publisher = {IEEE},
	Title = {Gradient-based learning applied to document recognition},
	Volume = {86},
	Year = {1998}}

@article{bengio1994learning,
	Author = {Bengio, Yoshua and Simard, Patrice and Frasconi, Paolo},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {IEEE transactions on neural networks},
	Number = {2},
	Pages = {157--166},
	Publisher = {IEEE},
	Title = {Learning long-term dependencies with gradient descent is difficult},
	Volume = {5},
	Year = {1994}}

@inproceedings{eklund2011relational,
  title={Relational Annotation of Scientific Medical Corpora},
  author={Eklund, Ann-Marie},
  booktitle={LOUHI 2011 Third International Workshop on Health Document Text Mining and Information Analysis},
  pages={27},
  year={2011}
}

@inproceedings{rosario2005multi,
  title={Multi-way relation classification: application to protein-protein interactions},
  author={Rosario, Barbara and Hearst, Marti A},
  booktitle={Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing},
  pages={732--739},
  year={2005},
  organization={Association for Computational Linguistics}
}

@article{hochreiter1997long,
	Author = {Hochreiter, Sepp and Schmidhuber, J{\"u}rgen},
	Date-Added = {2018-02-23 08:44:55 +0000},
	Date-Modified = {2018-02-23 08:44:55 +0000},
	Journal = {Neural computation},
	Number = {8},
	Pages = {1735--1780},
	Publisher = {MIT Press},
	Title = {Long short-term memory},
	Volume = {9},
	Year = {1997}}

@inproceedings{doddington2004automatic,
  title={The Automatic Content Extraction (ACE) Program-Tasks, Data, and Evaluation.},
  author={Doddington, George R and Mitchell, Alexis and Przybocki, Mark A and Ramshaw, Lance A and Strassel, Stephanie and Weischedel, Ralph M},
  booktitle={LREC},
  volume={2},
  pages={1},
  year={2004}
}

@inproceedings{hendrickx2009semeval,
  title={Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals},
  author={Hendrickx, Iris and Kim, Su Nam and Kozareva, Zornitsa and Nakov, Preslav and {\'O} S{\'e}aghdha, Diarmuid and Pad{\'o}, Sebastian and Pennacchiotti, Marco and Romano, Lorenza and Szpakowicz, Stan},
  booktitle={Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions},
  pages={94--99},
  year={2009},
  organization={Association for Computational Linguistics}
}

@inproceedings{nguyen2015relation,
  title={Relation extraction: Perspective from convolutional neural networks},
  author={Nguyen, Thien Huu and Grishman, Ralph},
  booktitle={Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing},
  pages={39--48},
  year={2015}
}

@inproceedings{yu2016pairwise,
  title={Pairwise relation classification with mirror instances and a combined convolutional neural network},
  author={Yu, Jianfei and Jiang, Jing},
  booktitle={Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  pages={2366--2377},
  year={2016}
}
  
@inproceedings{manabu1994word,
  title={Word sense disambiguation and text segmentation based on lexical cohesion},
  author={Manabu, Okumura and Takeo, Honda},
  booktitle={Proceedings of the 15th conference on Computational linguistics-Volume 2},
  pages={755--761},
  year={1994},
  organization={Association for Computational Linguistics}
}

@inproceedings{gabor2018semeval,
  title={Semeval-2018 Task 7: Semantic relation extraction and classification in scientific papers},
  author={G{\'a}bor, Kata and Buscaldi, Davide and Schumann, Anne-Kathrin and QasemiZadeh, Behrang and Zargayouna, Haifa and Charnois, Thierry},
  booktitle={Proceedings of The 12th International Workshop on Semantic Evaluation},
  pages={679--688},
  year={2018}
}

@inproceedings{pratap2018talla,
  title={Talla at SemEval-2018 Task 7: Hybrid Loss Optimization for Relation Classification using Convolutional Neural Networks},
  author={Pratap, Bhanu and Shank, Daniel and Ositelu, Oladipo and Galbraith, Byron},
  booktitle={Proceedings of The 12th International Workshop on Semantic Evaluation},
  pages={863--867},
  year={2018}
}

@article{xu2015semantic,
  title={Semantic relation classification via convolutional neural networks with simple negative sampling},
  author={Xu, Kun and Feng, Yansong and Huang, Songfang and Zhao, Dongyan},
  journal={arXiv preprint arXiv:1506.07650},
  year={2015}
}

@inproceedings{flekova2016supersense,
  title={Supersense embeddings: A unified model for supersense interpretation, prediction, and utilization},
  author={Flekova, Lucie and Gurevych, Iryna},
  booktitle={Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  volume={1},
  pages={2029--2041},
  year={2016}
}

@article{rotsztejn2018eth,
  title={ETH-DS3Lab at SemEval-2018 Task 7: Effectively Combining Recurrent and Convolutional Neural Networks for Relation Classification and Extraction},
  author={Rotsztejn, Jonathan and Hollenstein, Nora and Zhang, Ce},
  journal={arXiv preprint arXiv:1804.02042},
  year={2018}
}

@inproceedings{hettinger2018claire,
  title={ClaiRE at SemEval-2018 Task 7: Classification of Relations using Embeddings},
  author={Hettinger, Lena and Dallmann, Alexander and Zehe, Albin and Niebler, Thomas and Hotho, Andreas},
  booktitle={Proceedings of The 12th International Workshop on Semantic Evaluation},
  pages={836--841},
  year={2018}
}

@inproceedings{jin2018medg,
  title={MIT-MEDG at SemEval-2018 Task 7: Semantic Relation Classification via Convolution Neural Network},
  author={Jin, Di and Dernoncourt, Franck and Sergeeva, Elena and McDermott, Matthew and Chauhan, Geeticka},
  booktitle={Proceedings of The 12th International Workshop on Semantic Evaluation},
  pages={798--804},
  year={2018}
}

@inproceedings{luan2018uwnlp,
  title={The UWNLP system at SemEval-2018 Task 7: Neural Relation Extraction Model with Selectively Incorporated Concept Embeddings},
  author={Luan, Yi and Ostendorf, Mari and Hajishirzi, Hannaneh},
  booktitle={Proceedings of The 12th International Workshop on Semantic Evaluation},
  pages={788--792},
  year={2018}
}

@article{lee2017semeval,
  title={MIT at SemEval-2017 task 10: relation extraction with convolutional neural networks},
  author={Lee, Ji Young and Dernoncourt, Franck and Szolovits, Peter},
  journal={arXiv preprint arXiv:1704.01523},
  year={2017}
}

@inproceedings{yin2018ircms,
  title={IRCMS at SemEval-2018 Task 7: Evaluating a basic CNN Method and Traditional Pipeline Method for Relation Classification},
  author={Yin, Zhongbo and Luo, Zhunchen and Wei, Luo and Bin, Mao and Changhai, Tian and Yuming, Ye and Shuai, Wu},
  booktitle={Proceedings of The 12th International Workshop on Semantic Evaluation},
  pages={811--815},
  year={2018}
}

@inproceedings{bollacker2008freebase,
  title={Freebase: a collaboratively created graph database for structuring human knowledge},
  author={Bollacker, Kurt and Evans, Colin and Paritosh, Praveen and Sturge, Tim and Taylor, Jamie},
  booktitle={Proceedings of the 2008 ACM SIGMOD international conference on Management of data},
  pages={1247--1250},
  year={2008},
  organization={AcM}
}

@article{lehmann2015dbpedia,
  title={DBpedia--a large-scale, multilingual knowledge base extracted from Wikipedia},
  author={Lehmann, Jens and Isele, Robert and Jakob, Max and Jentzsch, Anja and Kontokostas, Dimitris and Mendes, Pablo N and Hellmann, Sebastian and Morsey, Mohamed and Van Kleef, Patrick and Auer, S{\"o}ren and others},
  journal={Semantic Web},
  volume={6},
  number={2},
  pages={167--195},
  year={2015},
  publisher={IOS Press}
}

@inproceedings{schuhmacher2014knowledge,
  title={Knowledge-based graph document modeling},
  author={Schuhmacher, Michael and Ponzetto, Simone Paolo},
  booktitle={Proceedings of the 7th ACM international conference on Web search and data mining},
  pages={543--552},
  year={2014},
  organization={ACM}
}

@inproceedings{trouillon2016complex,
  title={Complex embeddings for simple link prediction},
  author={Trouillon, Th{\'e}o and Welbl, Johannes and Riedel, Sebastian and Gaussier, {\'E}ric and Bouchard, Guillaume},
  booktitle={International Conference on Machine Learning},
  pages={2071--2080},
  year={2016}
}

@inproceedings{bordes2013translating,
  title={Translating embeddings for modeling multi-relational data},
  author={Bordes, Antoine and Usunier, Nicolas and Garcia-Duran, Alberto and Weston, Jason and Yakhnenko, Oksana},
  booktitle={Advances in neural information processing systems},
  pages={2787--2795},
  year={2013}
}

@inproceedings{trouillon2016complex,
  title={Complex embeddings for simple link prediction},
  author={Trouillon, Th{\'e}o and Welbl, Johannes and Riedel, Sebastian and Gaussier, {\'E}ric and Bouchard, Guillaume},
  booktitle={International Conference on Machine Learning},
  pages={2071--2080},
  year={2016}
}

@inproceedings{nickel2011three,
  title={A Three-Way Model for Collective Learning on Multi-Relational Data.},
  author={Nickel, Maximilian and Tresp, Volker and Kriegel, Hans-Peter},
  booktitle={ICML},
  volume={11},
  pages={809--816},
  year={2011}
}

@inproceedings{wang2014knowledge,
  title={Knowledge Graph Embedding by Translating on Hyperplanes.},
  author={Wang, Zhen and Zhang, Jianwen and Feng, Jianlin and Chen, Zheng},
  booktitle={AAAI},
  volume={14},
  pages={1112--1119},
  year={2014}
}

@article{liu2017analogical,
  title={Analogical inference for multi-relational embeddings},
  author={Liu, Hanxiao and Wu, Yuexin and Yang, Yiming},
  journal={arXiv preprint arXiv:1705.02426},
  year={2017}
}

@inproceedings{riedel2013relation,
  title={Relation extraction with matrix factorization and universal schemas},
  author={Riedel, Sebastian and Yao, Limin and McCallum, Andrew and Marlin, Benjamin M},
  booktitle={Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  pages={74--84},
  year={2013}
}

@inproceedings{wang2014knowledge,
  title={Knowledge graph and text jointly embedding},
  author={Wang, Zhen and Zhang, Jianwen and Feng, Jianlin and Chen, Zheng},
  booktitle={Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)},
  pages={1591--1601},
  year={2014}
}

@inproceedings{zhao2015representation,
  title={Representation Learning for Measuring Entity Relatedness with Rich Information.},
  author={Zhao, Yu and Liu, Zhiyuan and Sun, Maosong},
  booktitle={IJCAI},
  pages={1412--1418},
  year={2015}
}

@inproceedings{guo2015semantically,
  title={Semantically smooth knowledge graph embedding},
  author={Guo, Shu and Wang, Quan and Wang, Bin and Wang, Lihong and Guo, Li},
  booktitle={Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  volume={1},
  pages={84--94},
  year={2015}
}

@article{chang2014typed,
  title={Typed tensor decomposition of knowledge bases for relation extraction},
  author={Chang, Kai-Wei and Yih, Scott Wen-tau and Yang, Bishan and Meek, Chris},
  year={2014}
}

@inproceedings{wang2015knowledge,
  title={Knowledge Base Completion Using Embeddings and Rules.},
  author={Wang, Quan and Wang, Bin and Guo, Li and others},
  booktitle={IJCAI},
  pages={1859--1866},
  year={2015}
}

@inproceedings{rocktaschel2015injecting,
  title={Injecting logical background knowledge into embeddings for relation extraction},
  author={Rockt{\"a}schel, Tim and Singh, Sameer and Riedel, Sebastian},
  booktitle={Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  pages={1119--1129},
  year={2015}
}

@inproceedings{klein2017improving,
  title={Improving neural knowledge base completion with cross-lingual projections},
  author={Klein, Patrick and Ponzetto, Simone Paolo and Glava{\v{s}}, Goran},
  year={2017},
  organization={Association for Computational Linguistics}
}

@inproceedings{xie2016representation,
  title={Representation Learning of Knowledge Graphs with Hierarchical Types.},
  author={Xie, Ruobing and Liu, Zhiyuan and Sun, Maosong},
  booktitle={IJCAI},
  pages={2965--2971},
  year={2016}
}

@article{yaghoobzadeh2016corpus,
  title={Corpus-level fine-grained entity typing using contextual information},
  author={Yaghoobzadeh, Yadollah and Sch{\"u}tze, Hinrich},
  journal={arXiv preprint arXiv:1606.07901},
  year={2016}
}

@inproceedings{ji2015knowledge,
  title={Knowledge graph embedding via dynamic mapping matrix},
  author={Ji, Guoliang and He, Shizhu and Xu, Liheng and Liu, Kang and Zhao, Jun},
  booktitle={Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  volume={1},
  pages={687--696},
  year={2015}
}

@article{humphreys1998unified,
  title={The unified medical language system: an informatics research collaboration},
  author={Humphreys, Betsy L and Lindberg, Donald AB and Schoolman, Harold M and Barnett, G Octo},
  journal={Journal of the American Medical Informatics Association},
  volume={5},
  number={1},
  pages={1--11},
  year={1998},
  publisher={BMJ Group BMA House, Tavistock Square, London, WC1H 9JR}
}

@inproceedings{wang2014medical,
  title={Medical relation extraction with manifold models},
  author={Wang, Chang and Fan, James},
  booktitle={Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  volume={1},
  pages={828--838},
  year={2014}
}

@inproceedings{han2018neural,
  title={Neural knowledge acquisition via mutual attention between knowledge graph and text},
  author={Han, Xu and Liu, Zhiyuan and Sun, Maosong},
  booktitle={Thirty-Second AAAI Conference on Artificial Intelligence},
  year={2018}
}

@inproceedings{zeng2015distant,
  title={Distant supervision for relation extraction via piecewise convolutional neural networks},
  author={Zeng, Daojian and Liu, Kang and Chen, Yubo and Zhao, Jun},
  booktitle={Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  pages={1753--1762},
  year={2015}
}

@inproceedings{mintz2009distant,
  title={Distant supervision for relation extraction without labeled data},
  author={Mintz, Mike and Bills, Steven and Snow, Rion and Jurafsky, Dan},
  booktitle={Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2},
  pages={1003--1011},
  year={2009},
  organization={Association for Computational Linguistics}
}

@inproceedings{riedel2010modeling,
  title={Modeling relations and their mentions without labeled text},
  author={Riedel, Sebastian and Yao, Limin and McCallum, Andrew},
  booktitle={Joint European Conference on Machine Learning and Knowledge Discovery in Databases},
  pages={148--163},
  year={2010},
  organization={Springer}
}

@inproceedings{hoffmann2011knowledge,
  title={Knowledge-based weak supervision for information extraction of overlapping relations},
  author={Hoffmann, Raphael and Zhang, Congle and Ling, Xiao and Zettlemoyer, Luke and Weld, Daniel S},
  booktitle={Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1},
  pages={541--550},
  year={2011},
  organization={Association for Computational Linguistics}
}

@inproceedings{surdeanu2012multi,
  title={Multi-instance multi-label learning for relation extraction},
  author={Surdeanu, Mihai and Tibshirani, Julie and Nallapati, Ramesh and Manning, Christopher D},
  booktitle={Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning},
  pages={455--465},
  year={2012},
  organization={Association for Computational Linguistics}
}

@inproceedings{lin2016neural,
  title={Neural relation extraction with selective attention over instances},
  author={Lin, Yankai and Shen, Shiqi and Liu, Zhiyuan and Luan, Huanbo and Sun, Maosong},
  booktitle={Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  volume={1},
  pages={2124--2133},
  year={2016}
}

@article{wang2017knowledge,
  title={Knowledge graph embedding: A survey of approaches and applications},
  author={Wang, Quan and Mao, Zhendong and Wang, Bin and Guo, Li},
  journal={IEEE Transactions on Knowledge and Data Engineering},
  volume={29},
  number={12},
  pages={2724--2743},
  year={2017},
  publisher={IEEE}
}

@inproceedings{kazemi2018simple,
  title={SimplE embedding for link prediction in knowledge graphs},
  author={Kazemi, Seyed Mehran and Poole, David},
  booktitle={Advances in Neural Information Processing Systems},
  pages={4289--4300},
  year={2018}
}

@inproceedings{zhou2016attention,
  title={Attention-based bidirectional long short-term memory networks for relation classification},
  author={Zhou, Peng and Shi, Wei and Tian, Jun and Qi, Zhenyu and Li, Bingchen and Hao, Hongwei and Xu, Bo},
  booktitle={Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  volume={2},
  pages={207--212},
  year={2016}
}

@article{weston2013connecting,
  title={Connecting language and knowledge bases with embedding models for relation extraction},
  author={Weston, Jason and Bordes, Antoine and Yakhnenko, Oksana and Usunier, Nicolas},
  journal={arXiv preprint arXiv:1307.7973},
  year={2013}
}

@inproceedings{wang2014knowledge,
  title={Knowledge graph and text jointly embedding},
  author={Wang, Zhen and Zhang, Jianwen and Feng, Jianlin and Chen, Zheng},
  booktitle={Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)},
  pages={1591--1601},
  year={2014}
}

@article{american2013rise,
  title={The rise of open access},
  journal={Science},
  author={Randall Munroe},
  volume={342},
  number={6154},
  pages={58–-59},
  year={2013}
}

@inproceedings{soldaini2016quickumls,
  title={Quickumls: a fast, unsupervised approach for medical concept extraction},
  author={Soldaini, Luca and Goharian, Nazli},
  booktitle={MedIR workshop, sigir},
  year={2016}
}

@inproceedings{min2013distant,
  title={Distant supervision for relation extraction with an incomplete knowledge base},
  author={Min, Bonan and Grishman, Ralph and Wan, Li and Wang, Chang and Gondek, David},
  booktitle={Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  pages={777--782},
  year={2013}
}

@inproceedings{du2018multi,
  title={Multi-Level Structured Self-Attentions for Distantly Supervised Relation Extraction},
  author={Du, Jinhua and Han, Jingguang and Way, Andy and Wan, Dadong},
  booktitle={Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  pages={2216--2225},
  year={2018}
}

@inproceedings{lin2017neural,
  title={Neural relation extraction with multi-lingual attention},
  author={Lin, Yankai and Liu, Zhiyuan and Sun, Maosong},
  booktitle={Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={34--43},
  year={2017}
}

@inproceedings{ji2017distant,
  title={Distant supervision for relation extraction with sentence-level attention and entity descriptions},
  author={Ji, Guoliang and Liu, Kang and He, Shizhu and Zhao, Jun},
  booktitle={Thirty-First AAAI Conference on Artificial Intelligence},
  year={2017}
}

@inproceedings{vashishth2018reside,
  title={RESIDE: Improving Distantly-Supervised Neural Relation Extraction using Side Information},
  author={Vashishth, Shikhar and Joshi, Rishabh and Prayaga, Sai Suman and Bhattacharyya, Chiranjib and Talukdar, Partha},
  booktitle={Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  pages={1257--1266},
  year={2018}
}

@inproceedings{roller2015improving,
  title={Improving distant supervision using inference learning},
  author={Roller, RA and Agirre, E and Sorora, A and Stevenson, M},
  booktitle={Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing},
  year={2015},
  organization={Association for Computational Linguistics}
}

@article{soderland1999learning,
  title={Learning information extraction rules for semi-structured and free text},
  author={Soderland, Stephen},
  journal={Machine learning},
  volume={34},
  number={1-3},
  pages={233--272},
  year={1999},
  publisher={Springer}
}

@inproceedings{pennington2014glove,
  title={Glove: Global vectors for word representation},
  author={Pennington, Jeffrey and Socher, Richard and Manning, Christopher},
  booktitle={Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)},
  pages={1532--1543},
  year={2014}
}

@article{bojanowski2017enriching,
  title={Enriching word vectors with subword information},
  author={Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas},
  journal={Transactions of the Association for Computational Linguistics},
  volume={5},
  pages={135--146},
  year={2017},
  publisher={MIT Press}
}

@inproceedings{quirk2017distant,
  title={Distant Supervision for Relation Extraction beyond the Sentence Boundary},
  author={Quirk, Chris and Poon, Hoifung},
  booktitle={Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers},
  pages={1171--1182},
  year={2017}
}

@inproceedings{das2017chains,
  title={Chains of Reasoning over Entities, Relations, and Text using Recurrent Neural Networks},
  author={Das, Rajarshi and Neelakantan, Arvind and Belanger, David and McCallum, Andrew},
  booktitle={Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers},
  pages={132--141},
  year={2017}
}

@inproceedings{groth2018open,
  title={Open Information Extraction on Scientific Text: An Evaluation},
  author={Groth, Paul and Lauruhn, Mike and Scerri, Antony and Daniel Jr, Ron},
  booktitle={Proceedings of the 27th International Conference on Computational Linguistics},
  pages={3414--3423},
  year={2018}
}

@inproceedings{do-etal-2012-joint,
    title = "Joint Inference for Event Timeline Construction",
    author = "Do, Quang  and
      Lu, Wei  and
      Roth, Dan",
    booktitle = "Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning",
    month = jul,
    year = "2012",
    address = "Jeju Island, Korea",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D12-1062",
    pages = "677--687",
}

@inproceedings{do-roth-2010-constraints,
    title = "Constraints Based Taxonomic Relation Classification",
    author = "Do, Quang  and
      Roth, Dan",
    booktitle = "Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2010",
    address = "Cambridge, MA",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D10-1107",
    pages = "1099--1109",
}

@inproceedings{lao2012reading,
  title={Reading the web with learned syntactic-semantic inference rules},
  author={Lao, Ni and Subramanya, Amarnag and Pereira, Fernando and Cohen, William W},
  booktitle={Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning},
  pages={1017--1026},
  year={2012},
  organization={Association for Computational Linguistics}
}

@inproceedings{lin-etal-2018-multi,
    title = "Multi-Hop Knowledge Graph Reasoning with Reward Shaping",
    author = "Lin, Xi Victoria  and
      Socher, Richard  and
      Xiong, Caiming",
    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",
    month = oct # "-" # nov,
    year = "2018",
    address = "Brussels, Belgium",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D18-1362",
    doi = "10.18653/v1/D18-1362",
    pages = "3243--3253"
}

@inproceedings{xiong-etal-2017-deeppath,
    title = "{D}eep{P}ath: A Reinforcement Learning Method for Knowledge Graph Reasoning",
    author = "Xiong, Wenhan  and
      Hoang, Thien  and
      Wang, William Yang",
    booktitle = "Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing",
    month = sep,
    year = "2017",
    address = "Copenhagen, Denmark",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D17-1060",
    doi = "10.18653/v1/D17-1060",
    pages = "564--573"
}

@article{das2017go,
  title={Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning},
  author={Das, Rajarshi and Dhuliawala, Shehzaad and Zaheer, Manzil and Vilnis, Luke and Durugkar, Ishan and Krishnamurthy, Akshay and Smola, Alex and McCallum, Andrew},
  journal={arXiv preprint arXiv:1711.05851},
  year={2017}
}

@inproceedings{finkel2005incorporating,
  title={Incorporating non-local information into information extraction systems by gibbs sampling},
  author={Finkel, Jenny Rose and Grenager, Trond and Manning, Christopher},
  booktitle={Proceedings of the 43rd annual meeting on association for computational linguistics},
  pages={363--370},
  year={2005},
  organization={Association for Computational Linguistics}
}

@inproceedings{feng2017effective,
  title={Effective Deep Memory Networks for Distant Supervised Relation Extraction.},
  author={Feng, Xiaocheng and Guo, Jiang and Qin, Bing and Liu, Ting and Liu, Yongjie},
  booktitle={IJCAI},
  pages={4002--4008},
  year={2017}
}

@inproceedings{he2018see,
  title={SEE: Syntax-aware entity embedding for neural relation extraction},
  author={He, Zhengqiu and Chen, Wenliang and Li, Zhenghua and Zhang, Meishan and Zhang, Wei and Zhang, Min},
  booktitle={Thirty-Second AAAI Conference on Artificial Intelligence},
  year={2018}
}

@inproceedings{bast2014easy,
  title={Easy access to the freebase dataset},
  author={Bast, Hannah and B{\"a}urle, Florian and Buchhold, Bj{\"o}rn and Hau{\ss}mann, Elmar},
  booktitle={Proceedings of the 23rd International Conference on World Wide Web},
  pages={95--98},
  year={2014},
  organization={ACM}
}

@article{article,
author = {Gabrilovich, Evgeniy and Ringgaard, Michael and Subramanya, Amarnag},
year = {2013},
month = {06},
pages = {},
title = {FACC1: Freebase annotation of ClueWeb corpora, Version 1 (Release date 2013-06-26, Format version 1, Correction level 0)}
}

@article{alt2019fine,
  title={Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction},
  author={Alt, Christoph and H{\"u}bner, Marc and Hennig, Leonhard},
  journal={arXiv preprint arXiv:1906.08646},
  year={2019}
}

@article{jat2018improving,
  title={Improving distantly supervised relation extraction using word and entity based attention},
  author={Jat, Sharmistha and Khandelwal, Siddhesh and Talukdar, Partha},
  journal={arXiv preprint arXiv:1804.06987},
  year={2018}
}

@inproceedings{sun2019leveraging,
  title={Leveraging 2-hop Distant Supervision from Table Entity Pairs for Relation Extraction},
  author={Sun, Huan and others},
  booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  pages={410--420},
  year={2019}
}

@inproceedings{zeng2017incorporating,
  title={Incorporating Relation Paths in Neural Relation Extraction},
  author={Zeng, Wenyuan and Lin, Yankai and Liu, Zhiyuan and Sun, Maosong},
  booktitle={Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  pages={1768--1777},
  year={2017}
}

@inproceedings{neelakantan-etal-2015-compositional,
    title = "Compositional Vector Space Models for Knowledge Base Completion",
    author = "Neelakantan, Arvind  and
      Roth, Benjamin  and
      McCallum, Andrew",
    booktitle = "Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)",
    month = jul,
    year = "2015",
    address = "Beijing, China",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/P15-1016",
    doi = "10.3115/v1/P15-1016",
    pages = "156--166",
}

@inproceedings{dai2019incorporating,
  title={Incorporating Chains of Reasoning over Knowledge Graph for Distantly Supervised Biomedical Knowledge Acquisition},
  author={Dai, Qin and Inoue, Naoya and Reisert, Paul and Ryo, Takahashi and Inui, Kentaro},
  booktitle={Proceedings of the 33nd Pacific Asia Conference on Language, Information and Computation (PACLIC33)},
  year = "2019",
  address = "Hakodate, Japan",
  publisher = "Waseda Institute for the Study of Language and Information",
  url = "http://jaslli.org/files/proceedings/03_paclic33_postconf.pdf",
  pages = "19--28",
}

@inproceedings{jia2019arnor,
  title={ARNOR: attention regularization based noise reduction for distant supervision relation classification},
  author={Jia, Wei and Dai, Dai and Xiao, Xinyan and Wu, Hua},
  booktitle={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  pages={1399--1408},
  year={2019}
}

@inproceedings{neumann-etal-2019-scispacy,
    title = "{S}cispa{C}y: {F}ast and {R}obust {M}odels for {B}iomedical {N}atural {L}anguage {P}rocessing",
    author = "Neumann, Mark  and
      King, Daniel  and
      Beltagy, Iz  and
      Ammar, Waleed",
    booktitle = "Proceedings of the 18th BioNLP Workshop and Shared Task",
    month = aug,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/W19-5034",
    doi = "10.18653/v1/W19-5034",
    pages = "319--327",
    eprint = {arXiv:1902.07669},
    abstract = "Despite recent advances in natural language processing, many statistical models for processing text perform extremely poorly under domain shift. Processing biomedical and clinical text is a critically important application area of natural language processing, for which there are few robust, practical, publicly available models. This paper describes scispaCy, a new Python library and models for practical biomedical/scientific text processing, which heavily leverages the spaCy library. We detail the performance of two packages of models released in scispaCy and demonstrate their robustness on several tasks and datasets. Models and code are available at https://allenai.github.io/scispacy/.",
}

@inproceedings{han2018hierarchical,
  title={Hierarchical relation extraction with coarse-to-fine grained attention},
  author={Han, Xu and Yu, Pengfei and Liu, Zhiyuan and Sun, Maosong and Li, Peng},
  booktitle={Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  pages={2236--2245},
  year={2018}
}

@inproceedings{liu2017soft,
  title={A soft-label method for noise-tolerant distantly supervised relation extraction},
  author={Liu, Tianyu and Wang, Kexiang and Chang, Baobao and Sui, Zhifang},
  booktitle={Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  pages={1790--1795},
  year={2017}
}

@inproceedings{ye2019distant,
  title={Distant Supervision Relation Extraction with Intra-Bag and Inter-Bag Attentions},
  author={Ye, Zhi-Xiu and Ling, Zhen-Hua},
  booktitle={Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  pages={2810--2819},
  year={2019}
}

@inproceedings{brunato2018sentence,
  title={Is this Sentence Difficult? Do you Agree?},
  author={Brunato, Dominique and De Mattei, Lorenzo and Dell’Orletta, Felice and Iavarone, Benedetta and Venturi, Giulia},
  booktitle={Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  pages={2690--2699},
  year={2018}
}

@article{gabrilovich2013facc1,
author = {Gabrilovich, Evgeniy and Ringgaard, Michael and Subramanya, Amarnag},
year = {2013},
month = {06},
pages = {},
title = {FACC1: Freebase annotation of ClueWeb corpora, Version 1 (Release date 2013-06-26, Format version 1, Correction level 0)}
}

@inproceedings{neelakantan2015compositional,
  title={Compositional Vector Space Models for Knowledge Base Completion},
  author={Neelakantan, Arvind and Roth, Benjamin and McCallum, Andrew},
  booktitle={Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  pages={156--166},
  year={2015}
}

@inproceedings{toutanova2015representing,
  title={Representing text for joint embedding of text and knowledge bases},
  author={Toutanova, Kristina and Chen, Danqi and Pantel, Patrick and Poon, Hoifung and Choudhury, Pallavi and Gamon, Michael},
  booktitle={Proceedings of the 2015 conference on empirical methods in natural language processing},
  pages={1499--1509},
  year={2015}
}

@inproceedings{das2017question,
  title={Question Answering on Knowledge Bases and Text using Universal Schema and Memory Networks},
  author={Das, Rajarshi and Zaheer, Manzil and Reddy, Siva and McCallum, Andrew},
  booktitle={Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  pages={358--365},
  year={2017}
}