  Hateful rhetoric is plaguing online discourse, fostering extreme societal
movements and possibly giving rise to real-world violence. A potential solution
to this growing global problem is citizen-generated counter speech where
citizens actively engage in hate-filled conversations to attempt to restore
civil non-polarized discourse. However, its actual effectiveness in curbing the
spread of hatred is unknown and hard to quantify. One major obstacle to
researching this question is a lack of large labeled data sets for training
automated classifiers to identify counter speech. Here we made use of a unique
situation in Germany where self-labeling groups engaged in organized online
hate and counter speech. We used an ensemble learning algorithm which pairs a
variety of paragraph embeddings with regularized logistic regression functions
to classify both hate and counter speech in a corpus of millions of relevant
tweets from these two groups. Our pipeline achieved macro F1 scores on out of
sample balanced test sets ranging from 0.76 to 0.97---accuracy in line and even
exceeding the state of the art. On thousands of tweets, we used crowdsourcing
to verify that the judgments made by the classifier are in close alignment with
human judgment. We then used the classifier to discover hate and counter speech
in more than 135,000 fully-resolved Twitter conversations occurring from 2013
to 2018 and study their frequency and interaction. Altogether, our results
highlight the potential of automated methods to evaluate the impact of
coordinated counter speech in stabilizing conversations on social media.
