  Conversational systems have become increasingly popular as a way for humans
to interact with computers. To be able to provide intelligent responses,
conversational systems must correctly model the structure and semantics of a
conversation. We introduce the task of measuring semantic (in)coherence in a
conversation with respect to background knowledge, which relies on the
identification of semantic relations between concepts introduced during a
conversation. We propose and evaluate graph-based and machine learning-based
approaches for measuring semantic coherence using knowledge graphs, their
vector space embeddings and word embedding models, as sources of background
knowledge. We demonstrate how these approaches are able to uncover different
coherence patterns in conversations on the Ubuntu Dialogue Corpus.
