  Continuous sentence embeddings using recurrent neural networks (RNNs), where
variable-length sentences are encoded into fixed-dimensional vectors, are often
the main building blocks of architectures applied to language tasks such as
dialogue generation. While it is known that those embeddings are able to learn
some structures of language (e.g. grammar) in a purely data-driven manner,
there is very little work on the objective evaluation of their ability to cover
the whole language space and to generalize to sentences outside the language
bias of the training data. Using a manually designed context-free grammar (CFG)
to generate a large-scale dataset of sentences related to the content of
realistic 3D indoor scenes, we evaluate the language coverage and
generalization abilities of the most common continuous sentence embeddings
based on RNNs. We also propose a new embedding method based on arithmetic
coding, AriEL, that is not data-driven and that efficiently encodes in
continuous space any sentence from the CFG. We find that RNN-based embeddings
underfit the training data and cover only a small subset of the language
defined by the CFG. They also fail to learn the underlying CFG and generalize
to unbiased sentences from that same CFG. We found that AriEL provides an
insightful baseline.
