  Dialogue models trained on human conversations inadvertently learn to
generate toxic responses. In addition to producing explicitly offensive
utterances, these models can also implicitly insult a group or individual by
aligning themselves with an offensive statement. To better understand the
dynamics of contextually offensive language, we investigate the stance of
dialogue model responses in offensive Reddit conversations. Specifically, we
create ToxiChat, a crowd-annotated dataset of 2,000 Reddit threads and model
responses labeled with offensive language and stance. Our analysis reveals that
42% of human responses agree with toxic comments, whereas only 13% agree with
safe comments. This undesirable behavior is learned by neural dialogue models,
such as DialoGPT, which we show are two times more likely to agree with
offensive comments. To enable automatic detection of offensive language, we
fine-tuned transformer-based classifiers on ToxiChat that achieve 0.71 F1 for
offensive labels and 0.53 Macro-F1 for stance labels. Finally, we quantify the
effectiveness of controllable text generation (CTG) methods to mitigate the
tendency of neural dialogue models to agree with offensive comments. Compared
to the baseline, our best CTG model achieves a 19% reduction in agreement with
offensive comments and produces 29% fewer offensive replies. Our work
highlights the need for further efforts to characterize and analyze
inappropriate behavior in dialogue models, in order to help make them safer.
Our code and corpus are available at https://github.com/abaheti95/ToxiChat .
