  The spread of false rumours during emergencies can jeopardise the well-being
of citizens as they are monitoring the stream of news from social media to stay
abreast of the latest updates. In this paper, we describe the methodology we
have developed within the PHEME project for the collection and sampling of
conversational threads, as well as the tool we have developed to facilitate the
annotation of these threads so as to identify rumourous ones. We describe the
annotation task conducted on threads collected during the 2014 Ferguson unrest
and we present and analyse our findings. Our results show that we can collect
effectively social media rumours and identify multiple rumours associated with
a range of stories that would have been hard to identify by relying on existing
techniques that need manual input of rumour-specific keywords.
