  With adversarial or otherwise normal prompts, existing large language models
(LLM) can be pushed to generate toxic discourses. One way to reduce the risk of
LLMs generating undesired discourses is to alter the training of the LLM. This
can be very restrictive due to demanding computation requirements. Other
methods rely on rule-based or prompt-based token elimination, which are limited
as they dismiss future tokens and the overall meaning of the complete
discourse. Here, we center detoxification on the probability that the finished
discourse is ultimately considered toxic. That is, at each point, we advise
against token selections proportional to how likely a finished text from this
point will be toxic. To this end, we formally extend the dead-end theory from
the recent reinforcement learning (RL) literature to also cover uncertain
outcomes. Our approach, called rectification, utilizes a separate but
significantly smaller model for detoxification, which can be applied to diverse
LLMs as long as they share the same vocabulary. Importantly, our method does
not require access to the internal representations of the LLM, but only the
token probability distribution at each decoding step. This is crucial as many
LLMs today are hosted in servers and only accessible through APIs. When applied
to various LLMs, including GPT-3, our approach significantly improves the
generated discourse compared to the base LLMs and other techniques in terms of
both the overall language and detoxification performance.
