  The wealth of structured (e.g. Wikidata) and unstructured data about the
world available today presents an incredible opportunity for tomorrow's
Artificial Intelligence. So far, integration of these two different modalities
is a difficult process, involving many decisions concerning how best to
represent the information so that it will be captured or useful, and
hand-labeling large amounts of data. DeepType overcomes this challenge by
explicitly integrating symbolic information into the reasoning process of a
neural network with a type system. First we construct a type system, and
second, we use it to constrain the outputs of a neural network to respect the
symbolic structure. We achieve this by reformulating the design problem into a
mixed integer problem: create a type system and subsequently train a neural
network with it. In this reformulation discrete variables select which
parent-child relations from an ontology are types within the type system, while
continuous variables control a classifier fit to the type system. The original
problem cannot be solved exactly, so we propose a 2-step algorithm: 1)
heuristic search or stochastic optimization over discrete variables that define
a type system informed by an Oracle and a Learnability heuristic, 2) gradient
descent to fit classifier parameters. We apply DeepType to the problem of
Entity Linking on three standard datasets (i.e. WikiDisamb30, CoNLL (YAGO), TAC
KBP 2010) and find that it outperforms all existing solutions by a wide margin,
including approaches that rely on a human-designed type system or recent deep
learning-based entity embeddings, while explicitly using symbolic information
lets it integrate new entities without retraining.
