  The parametric knowledge memorized by large language models (LLMs) becomes
outdated quickly. In-context editing (ICE) is currently the most effective
method for updating the knowledge of LLMs. Recent advancements involve
enhancing ICE by modifying the decoding strategy, obviating the need for
altering internal model structures or adjusting external prompts. However, this
enhancement operates across the entire sequence generation, encompassing a
plethora of non-critical tokens. In this work, we introduce $\textbf{A}$daptive
$\textbf{T}$oken $\textbf{Bias}$er ($\textbf{ATBias}$), a new decoding
technique designed to enhance ICE. It focuses on the tokens that are mostly
related to knowledge during decoding, biasing their logits by matching key
entities related to new and parametric knowledge. Experimental results show
that ATBias significantly enhances ICE performance, achieving up to a 32.3%
improvement over state-of-the-art ICE methods while incurring only half the
latency. ATBias not only improves the knowledge editing capabilities of ICE but
can also be widely applied to LLMs with negligible cost.
