  Social aspects of software projects become increasingly important for
research and practice. Different approaches analyze the sentiment of a
development team, ranging from simply asking the team to so-called sentiment
analysis on text-based communication. These sentiment analysis tools are
trained using pre-labeled data sets from different sources, including GitHub
and Stack Overflow. In this paper, we investigate if the labels of the
statements in the data sets coincide with the perception of potential members
of a software project team. Based on an international survey, we compare the
median perception of 94 participants with the pre-labeled data sets as well as
every single participant's agreement with the predefined labels. Our results
point to three remarkable findings: (1) Although the median values coincide
with the predefined labels of the data sets in 62.5% of the cases, we observe a
huge difference between the single participant's ratings and the labels; (2)
there is not a single participant who totally agrees with the predefined
labels; and (3) the data set whose labels are based on guidelines performs
better than the ad hoc labeled data set.
