  Collecting together microblogs representing opinions about the same topics
within the same timeframe is useful to a number of different tasks and
practitioners. A major question is how to evaluate the quality of such thematic
clusters. Here we create a corpus of microblog clusters from three different
domains and time windows and define the task of evaluating thematic coherence.
We provide annotation guidelines and human annotations of thematic coherence by
journalist experts. We subsequently investigate the efficacy of different
automated evaluation metrics for the task. We consider a range of metrics
including surface level metrics, ones for topic model coherence and text
generation metrics (TGMs). While surface level metrics perform well,
outperforming topic coherence metrics, they are not as consistent as TGMs. TGMs
are more reliable than all other metrics considered for capturing thematic
coherence in microblog clusters due to being less sensitive to the effect of
time windows.
