  We present a new method based on episodic Knowledge Graphs (eKGs) for
evaluating (multimodal) conversational agents in open domains. This graph is
generated by interpreting raw signals during conversation and is able to
capture the accumulation of knowledge over time. We apply structural and
semantic analysis of the resulting graphs and translate the properties into
qualitative measures. We compare these measures with existing automatic and
manual evaluation metrics commonly used for conversational agents. Our results
show that our Knowledge-Graph-based evaluation provides more qualitative
insights into interaction and the agent's behavior.
