  Large language models (LLMs) have shown incredible performance in completing
various real-world tasks. The current knowledge learning paradigm of LLMs is
mainly based on learning from examples, in which LLMs learn the internal rule
implicitly from a certain number of supervised examples. However, this learning
paradigm may not well learn those complicated rules, especially when the
training examples are limited. We are inspired that humans can learn the new
tasks or knowledge in another way by learning from rules. That is, humans can
learn new tasks or grasps new knowledge quickly and generalize well given only
a detailed rule and a few optional examples. Therefore, in this paper, we aim
to explore the feasibility of this new learning paradigm, which targets on
encoding rule-based knowledge into LLMs. We further propose rule distillation,
which first uses the strong in-context abilities of LLMs to extract the
knowledge from the textual rules, and then explicitly encode the knowledge into
the parameters of LLMs by learning from the above in-context signals produced
inside the model. Our experiments show that making LLMs learn from rules by our
method is much more efficient than example-based learning in both the sample
size and generalization ability. Warning: This paper may contain examples with
offensive content.
