  The rise of "jailbreak" attacks on language models has led to a flurry of
defenses aimed at preventing undesirable responses. We critically examine the
two stages of the defense pipeline: (i) defining what constitutes unsafe
outputs, and (ii) enforcing the definition via methods such as input processing
or fine-tuning. To test the efficacy of existing enforcement mechanisms, we
consider a simple and well-specified definition of unsafe outputs--outputs that
contain the word "purple". Surprisingly, existing fine-tuning and input
defenses fail on this simple problem, casting doubt on whether enforcement
algorithms can be robust for more complicated definitions. We find that real
safety benchmarks similarly test enforcement for a fixed definition. We hope
that future research can lead to effective/fast enforcement as well as high
quality definitions used for enforcement and evaluation.
