  To reduce the toxic degeneration in a pretrained Language Model (LM),
previous work on Language Model detoxification has focused on reducing the
toxicity of the generation itself (self-toxicity) without consideration of the
context. As a result, a type of implicit offensive language where the
generations support the offensive language in the context is ignored. Different
from the LM controlling tasks in previous work, where the desired attributes
are fixed for generation, the desired stance of the generation depends on the
offensiveness of the context. Therefore, we propose a novel control method to
do context-dependent detoxification with the stance taken into consideration.
We introduce meta prefixes to learn the contextualized stance control strategy
and to generate the stance control prefix according to the input context. The
generated stance prefix is then combined with the toxicity control prefix to
guide the response generation. Experimental results show that our proposed
method can effectively learn the context-dependent stance control strategies
while keeping a low self-toxicity of the underlying LM.
