  We study how to subvert language models from following the rules. We model
rule-following as inference in propositional Horn logic, a mathematical system
in which rules have the form "if $P$ and $Q$, then $R$" for some propositions
$P$, $Q$, and $R$. We prove that although transformers can faithfully abide by
such rules, maliciously crafted prompts can nevertheless mislead even
theoretically constructed models. Empirically, we find that attacks on our
theoretical models mirror popular attacks on large language models. Our work
suggests that studying smaller theoretical models can help understand the
behavior of large language models in rule-based settings like logical reasoning
and jailbreak attacks.
