  Despite the common use of rule-based tools for online content moderation,
human moderators still spend a lot of time monitoring them to ensure that they
work as intended. Based on surveys and interviews with Reddit moderators who
use AutoModerator, we identified the main challenges in reducing false
positives and false negatives of automated rules: not being able to estimate
the actual effect of a rule in advance and having difficulty figuring out how
the rules should be updated. To address these issues, we built ModSandbox, a
novel virtual sandbox system that detects possible false positives and false
negatives of a rule to be improved and visualizes which part of the rule is
causing issues. We conducted a user study with online content moderators,
finding that ModSandbox can support quickly finding possible false positives
and false negatives of automated rules and guide moderators to update those to
reduce future errors.
