  Conversational moderation of online communities is crucial to maintaining
civility for a constructive environment, but it is challenging to scale and
harmful to moderators. The inclusion of sophisticated natural language
generation modules as a force multiplier to aid human moderators is a
tantalizing prospect, but adequate evaluation approaches have so far been
elusive. In this paper, we establish a systematic definition of conversational
moderation effectiveness grounded on moderation literature and establish design
criteria for conducting realistic yet safe evaluation. We then propose a
comprehensive evaluation framework to assess models' moderation capabilities
independently of human intervention. With our framework, we conduct the first
known study of language models as conversational moderators, finding that
appropriately prompted models that incorporate insights from social science can
provide specific and fair feedback on toxic behavior but struggle to influence
users to increase their levels of respect and cooperation.
