  Compositionality has traditionally been understood as a major factor in
productivity of language and, more broadly, human cognition. Yet, recently,
some research started to question its status, showing that artificial neural
networks are good at generalization even without noticeable compositional
behavior. We argue that some of these conclusions are too strong and/or
incomplete. In the context of a two-agent communication game, we show that
compositionality indeed seems essential for successful generalization when the
evaluation is done on a proper dataset.
