  In this work, we systematically investigate how well current models of
coherence can capture aspects of text implicated in discourse organisation. We
devise two datasets of various linguistic alterations that undermine coherence
and test model sensitivity to changes in syntax and semantics. We furthermore
probe discourse embedding space and examine the knowledge that is encoded in
representations of coherence. We hope this study shall provide further insight
into how to frame the task and improve models of coherence assessment further.
Finally, we make our datasets publicly available as a resource for researchers
to use to test discourse coherence models.
