  Scarcity of labeled data is one of the most frequent problems faced in
machine learning. This is particularly true in relation extraction in text
mining, where large corpora of texts exists in many application domains, while
labeling of text data requires an expert to invest much time to read the
documents. Overall, state-of-the art models, like the convolutional neural
network used in this paper, achieve great results when trained on large enough
amounts of labeled data. However, from a practical point of view the question
arises whether this is the most efficient approach when one takes the manual
effort of the expert into account. In this paper, we report on an alternative
approach where we first construct a relation extraction model using distant
supervision, and only later make use of a domain expert to refine the results.
Distant supervision provides a mean of labeling data given known relations in a
knowledge base, but it suffers from noisy labeling. We introduce an active
learning based extension, that allows our neural network to incorporate expert
feedback and report on first results on a complex data set.
