  User-generated replies to hate speech are promising means to combat hatred,
but questions about whether they can stop incivility in follow-up conversations
linger. We argue that effective replies stop incivility from emerging in
follow-up conversations - replies that elicit more incivility are
counterproductive. This study introduces the task of predicting the incivility
of conversations following replies to hate speech. We first propose a metric to
measure conversation incivility based on the number of civil and uncivil
comments as well as the unique authors involved in the discourse. Our metric
approximates human judgments more accurately than previous metrics. We then use
the metric to evaluate the outcomes of replies to hate speech. A linguistic
analysis uncovers the differences in the language of replies that elicit
follow-up conversations with high and low incivility. Experimental results show
that forecasting incivility is challenging. We close with a qualitative
analysis shedding light into the most common errors made by the best model.
