  Automated ways to extract stance (denying vs. supporting opinions) from
conversations on social media are essential to advance opinion mining research.
Recently, there is a renewed excitement in the field as we see new models
attempting to improve the state-of-the-art. However, for training and
evaluating the models, the datasets used are often small. Additionally, these
small datasets have uneven class distributions, i.e., only a tiny fraction of
the examples in the dataset have favoring or denying stances, and most other
examples have no clear stance. Moreover, the existing datasets do not
distinguish between the different types of conversations on social media (e.g.,
replying vs. quoting on Twitter). Because of this, models trained on one event
do not generalize to other events.
  In the presented work, we create a new dataset by labeling stance in
responses to posts on Twitter (both replies and quotes) on controversial
issues. To the best of our knowledge, this is currently the largest
human-labeled stance dataset for Twitter conversations with over 5200 stance
labels. More importantly, we designed a tweet collection methodology that
favors the selection of denial-type responses. This class is expected to be
more useful in the identification of rumors and determining antagonistic
relationships between users. Moreover, we include many baseline models for
learning the stance in conversations and compare the performance of various
models. We show that combining data from replies and quotes decreases the
accuracy of models indicating that the two modalities behave differently when
it comes to stance learning.
