  The knowledge that humans hold about a problem often extends far beyond a set
of training data and output labels. While the success of deep learning mostly
relies on supervised training, important properties cannot be inferred
efficiently from end-to-end annotations alone, for example causal relations or
domain-specific invariances. We present a general technique to supplement
supervised training with prior knowledge expressed as relations between
training instances. We illustrate the method on the task of visual question
answering to exploit various auxiliary annotations, including relations of
equivalence and of logical entailment between questions. Existing methods to
use these annotations, including auxiliary losses and data augmentation, cannot
guarantee the strict inclusion of these relations into the model since they
require a careful balancing against the end-to-end objective. Our method uses
these relations to shape the embedding space of the model, and treats them as
strict constraints on its learned representations. In the context of VQA, this
approach brings significant improvements in accuracy and robustness, in
particular over the common practice of incorporating the constraints as a soft
regularizer. We also show that incorporating this type of prior knowledge with
our method brings consistent improvements, independently from the amount of
supervised data used. It demonstrates the value of an additional training
signal that is otherwise difficult to extract from end-to-end annotations
alone.
