  Crowdsourcing information constitutes an important aspect of
human-in-the-loop learning for researchers across multiple disciplines such as
AI, HCI, and social science. While using crowdsourced data for subjective tasks
is not new, eliciting useful insights from such data remains challenging due to
a variety of factors such as difficulty of the task, personal prejudices of the
human evaluators, lack of question clarity, etc. In this paper, we consider one
such subjective evaluation task, namely that of estimating experienced emotions
of distressed individuals who are conversing with a human listener in an online
coaching platform. We explore strategies to aggregate the evaluators choices,
and show that a simple voting consensus is as effective as an optimum
aggregation method for the task considered. Intrigued by how an objective
assessment would compare to the subjective evaluation of evaluators, we also
designed a machine learning algorithm to perform the same task. Interestingly,
we observed a machine learning algorithm that is not explicitly modeled to
characterize evaluators' subjectivity is as reliable as the human evaluation in
terms of assessing the most dominant experienced emotions.
