  Social media, especially Twitter, is being increasingly used for research
with predictive analytics. In social media studies, natural language processing
(NLP) techniques are used in conjunction with expert-based, manual and
qualitative analyses. However, social media data are unstructured and must
undergo complex manipulation for research use. The manual annotation is the
most resource and time-consuming process that multiple expert raters have to
reach consensus on every item, but is essential to create gold-standard
datasets for training NLP-based machine learning classifiers. To reduce the
burden of the manual annotation, yet maintaining its reliability, we devised a
crowdsourcing pipeline combined with active learning strategies. We
demonstrated its effectiveness through a case study that identifies job loss
events from individual tweets. We used Amazon Mechanical Turk platform to
recruit annotators from the Internet and designed a number of quality control
measures to assure annotation accuracy. We evaluated 4 different active
learning strategies (i.e., least confident, entropy, vote entropy, and
Kullback-Leibler divergence). The active learning strategies aim at reducing
the number of tweets needed to reach a desired performance of automated
classification. Results show that crowdsourcing is useful to create
high-quality annotations and active learning helps in reducing the number of
required tweets, although there was no substantial difference among the
strategies tested.
