  Modern toxic speech detectors are incompetent in recognizing disguised
offensive language, such as adversarial attacks that deliberately avoid known
toxic lexicons, or manifestations of implicit bias. Building a large annotated
dataset for such veiled toxicity can be very expensive. In this work, we
propose a framework aimed at fortifying existing toxic speech detectors without
a large labeled corpus of veiled toxicity. Just a handful of probing examples
are used to surface orders of magnitude more disguised offenses. We augment the
toxic speech detector's training data with these discovered offensive examples,
thereby making it more robust to veiled toxicity while preserving its utility
in detecting overt toxicity.
