  Neural networks for natural language reasoning have largely focused on
extractive, fact-based question-answering (QA) and common-sense inference.
However, it is also crucial to understand the extent to which neural networks
can perform relational reasoning and combinatorial generalization from natural
language---abilities that are often obscured by annotation artifacts and the
dominance of language modeling in standard QA benchmarks. In this work, we
present a novel benchmark dataset for language understanding that isolates
performance on relational reasoning. We also present a neural message-passing
baseline and show that this model, which incorporates a relational inductive
bias, is superior at combinatorial generalization compared to a traditional
recurrent neural network approach.
