  Warning: this paper contains content that maybe offensive or upsetting.
Recent research in Natural Language Processing (NLP) has advanced the
development of various toxicity detection models with the intention of
identifying and mitigating toxic language from existing systems. Despite the
abundance of research in this area, less attention has been given to
adversarial attacks that force the system to generate toxic language and the
defense against them. Existing work to generate such attacks is either based on
human-generated attacks which is costly and not scalable or, in case of
automatic attacks, the attack vector does not conform to human-like language,
which can be detected using a language model loss. In this work, we propose
attacks against conversational agents that are imperceptible, i.e., they fit
the conversation in terms of coherency, relevancy, and fluency, while they are
effective and scalable, i.e., they can automatically trigger the system into
generating toxic language. We then propose a defense mechanism against such
attacks which not only mitigates the attack but also attempts to maintain the
conversational flow. Through automatic and human evaluations, we show that our
defense is effective at avoiding toxic language generation even against
imperceptible toxicity triggers while the generated language fits the
conversation in terms of coherency and relevancy. Lastly, we establish the
generalizability of such a defense mechanism on language generation models
beyond conversational agents.
