  Recognizing visual entities in a natural language sentence and arranging them
in a 2D spatial layout require a compositional understanding of language and
space. This task of layout prediction is valuable in text-to-image synthesis as
it allows localized and controlled in-painting of the image. In this
comparative study it is shown that we can predict layouts from language
representations that implicitly or explicitly encode sentence syntax, if the
sentences mention similar entity-relationships to the ones seen during
training. To test compositional understanding, we collect a test set of
grammatically correct sentences and layouts describing compositions of entities
and relations that unlikely have been seen during training. Performance on this
test set substantially drops, showing that current models rely on correlations
in the training data and have difficulties in understanding the structure of
the input sentences. We propose a novel structural loss function that better
enforces the syntactic structure of the input sentence and show large
performance gains in the task of 2D spatial layout prediction conditioned on
text. The loss has the potential to be used in other generation tasks where a
tree-like structure underlies the conditioning modality. Code, trained models
and the USCOCO evaluation set are available via github.
