  Parameter-Efficient Fine-Tuning (PEFT) methods enable efficient adaptation of
Large Language Models (LLMs) to various downstream applications. However, the
effectiveness of the PEFT diminishes notably when downstream tasks require
accurate learning of factual knowledge. In this paper, we adopt a semantic
perspective to investigate this phenomenon, uncovering the reasons behind
PEFT's limitations in knowledge learning task. Our findings reveal that: (1)
PEFT presents a notable risk of pushing the model away from the intended
knowledge target; (2) multiple knowledge interfere with each other, and such
interference suppresses the learning and expression of knowledge features.
Based on these insights, we introduce a data filtering strategy to exclude data
that is detrimental to knowledge learning and a re-weighted learning strategy
to make the model attentive to semantic distance during knowledge learning.
Experimental results demonstrate the effectiveness of the proposed method on
open-source large language model, further validate the semantic challenge in
PEFT, thus paving the way for future research.
