  Recently, deep neural networks (DNNs) have achieved great success in
semantically challenging NLP tasks, yet it remains unclear whether DNN models
can capture compositional meanings, those aspects of meaning that have been
long studied in formal semantics. To investigate this issue, we propose a
Systematic Generalization testbed based on Natural language Semantics (SyGNS),
whose challenge is to map natural language sentences to multiple forms of
scoped meaning representations, designed to account for various semantic
phenomena. Using SyGNS, we test whether neural networks can systematically
parse sentences involving novel combinations of logical expressions such as
quantifiers and negation. Experiments show that Transformer and GRU models can
generalize to unseen combinations of quantifiers, negations, and modifiers that
are similar to given training instances in form, but not to the others. We also
find that the generalization performance to unseen combinations is better when
the form of meaning representations is simpler. The data and code for SyGNS are
publicly available at https://github.com/verypluming/SyGNS.
