  Recently research has started focusing on avoiding undesired effects that
come with content moderation, such as censorship and overblocking, when dealing
with hatred online. The core idea is to directly intervene in the discussion
with textual responses that are meant to counter the hate content and prevent
it from further spreading. Accordingly, automation strategies, such as natural
language generation, are beginning to be investigated. Still, they suffer from
the lack of sufficient amount of quality data and tend to produce
generic/repetitive responses. Being aware of the aforementioned limitations, we
present a study on how to collect responses to hate effectively, employing
large scale unsupervised language models such as GPT-2 for the generation of
silver data, and the best annotation strategies/neural architectures that can
be used for data filtering before expert validation/post-editing.
