  One characteristic that sets humans apart from modern learning-based computer
vision algorithms is the ability to acquire knowledge about the world and use
that knowledge to reason about the visual world. Humans can learn about the
characteristics of objects and the relationships that occur between them to
learn a large variety of visual concepts, often with few examples. This paper
investigates the use of structured prior knowledge in the form of knowledge
graphs and shows that using this knowledge improves performance on image
classification. We build on recent work on end-to-end learning on graphs,
introducing the Graph Search Neural Network as a way of efficiently
incorporating large knowledge graphs into a vision classification pipeline. We
show in a number of experiments that our method outperforms standard neural
network baselines for multi-label classification.
