  Undermining the impact of hateful content with informed and non-aggressive
responses, called counter narratives, has emerged as a possible solution for
having healthier online communities. Thus, some NLP studies have started
addressing the task of counter narrative generation. Although such studies have
made an effort to build hate speech / counter narrative (HS/CN) datasets for
neural generation, they fall short in reaching either high-quality and/or
high-quantity. In this paper, we propose a novel human-in-the-loop data
collection methodology in which a generative language model is refined
iteratively by using its own data from the previous loops to generate new
training samples that experts review and/or post-edit. Our experiments
comprised several loops including dynamic variations. Results show that the
methodology is scalable and facilitates diverse, novel, and cost-effective data
collection. To our knowledge, the resulting dataset is the only expert-based
multi-target HS/CN dataset available to the community.
