  The incremental sequence labeling task involves continuously learning new
classes over time while retaining knowledge of the previous ones. Our
investigation identifies two significant semantic shifts: E2O (where the model
mislabels an old entity as a non-entity) and O2E (where the model labels a
non-entity or old entity as a new entity). Previous research has predominantly
focused on addressing the E2O problem, neglecting the O2E issue. This
negligence results in a model bias towards classifying new data samples as
belonging to the new class during the learning process. To address these
challenges, we propose a novel framework, Incremental Sequential Labeling
without Semantic Shifts (IS3). Motivated by the identified semantic shifts (E2O
and O2E), IS3 aims to mitigate catastrophic forgetting in models. As for the
E2O problem, we use knowledge distillation to maintain the model's
discriminative ability for old entities. Simultaneously, to tackle the O2E
problem, we alleviate the model's bias towards new entities through debiased
loss and optimization levels. Our experimental evaluation, conducted on three
datasets with various incremental settings, demonstrates the superior
performance of IS3 compared to the previous state-of-the-art method by a
significant margin.The data, code, and scripts are publicly available at
https://github.com/zzz47zzz/codebase-for-incremental-learning-with-llm.
