  As large language models (LLMs) are widely deployed, targeted editing of
their knowledge has become a critical challenge. Recently, advancements in
model editing techniques, such as Rank-One Model Editing (ROME), have paved the
way for updating LLMs with new knowledge. However, the efficacy of these
methods varies across different types of knowledge. This study investigates the
capability of knowledge editing methods to incorporate new knowledge with
varying degrees of "perplexingness", a term we use to describe the initial
difficulty LLMs have in understanding new concepts. We begin by quantifying the
"perplexingness" of target knowledge using pre-edit conditional probabilities,
and assess the efficacy of edits through post-edit conditional probabilities.
Utilizing the widely-used CounterFact dataset, we find significant negative
correlations between the "perplexingness" of the new knowledge and the edit
efficacy across all 12 scenarios. To dive deeper into this phenomenon, we
introduce a novel dataset, HierarchyData, consisting of 99 hyponym-hypernym
pairs across diverse categories. Our analysis reveal that more abstract
concepts (hypernyms) tend to be more perplexing than their specific
counterparts (hyponyms). Further exploration into the influence of knowledge
hierarchy on editing outcomes indicates that knowledge positioned at higher
hierarchical levels is more challenging to modify in some scenarios. Our
research highlights a previously overlooked aspect of LLM editing: the variable
efficacy of editing methods in handling perplexing knowledge. By revealing how
hierarchical relationships can influence editing outcomes, our findings offer
new insights into the challenges of updating LLMs and pave the way for more
nuanced approaches to model editing in the future.
