  We investigate ways to compose complex concepts in texts from primitive ones
while grounding them in images. We propose Concept and Relation Graph (CRG),
which builds on top of constituency analysis and consists of recursively
combined concepts with predicate functions. Meanwhile, we propose a concept
composition neural network called Composer to leverage the CRG for visually
grounded concept learning. Specifically, we learn the grounding of both
primitive and all composed concepts by aligning them to images and show that
learning to compose leads to more robust grounding results, measured in
text-to-image matching accuracy. Notably, our model can model grounded concepts
forming at both the finer-grained sentence level and the coarser-grained
intermediate level (or word-level). Composer leads to pronounced improvement in
matching accuracy when the evaluation data has significant compound divergence
from the training data.
