  Crowdsourcing is a popular means to obtain labeled data at moderate costs,
for example for tweets, which can then be used in text mining tasks. To
alleviate the problem of low-quality labels in this context, multiple human
factors have been analyzed to identify and deal with workers who provide such
labels. However, one aspect that has been rarely considered is the inherent
difficulty of tweets to be labeled and how this affects the reliability of the
labels that annotators assign to such tweets. Therefore, we investigate in this
preliminary study this connection using a hierarchical sentiment labeling task
on Twitter. We find that there is indeed a relationship between both factors,
assuming that annotators have labeled some tweets before: labels assigned to
easy tweets are more reliable than those assigned to difficult tweets.
Therefore, training predictors on easy tweets enhances the performance by up to
6% in our experiment. This implies potential improvements for active learning
techniques and crowdsourcing.
