  Large Language Models (LLMs) are frequently updated due to data or
architecture changes to improve their performance. When updating models,
developers often focus on increasing overall performance metrics with less
emphasis on being compatible with previous model versions. However, users often
build a mental model of the functionality and capabilities of a particular
machine learning model they are interacting with. They have to adapt their
mental model with every update -- a draining task that can lead to user
dissatisfaction. In practice, fine-tuned downstream task adapters rely on
pretrained LLM base models. When these base models are updated, these
user-facing downstream task models experience instance regression or negative
flips -- previously correct instances are now predicted incorrectly. This
happens even when the downstream task training procedures remain identical. Our
work aims to provide seamless model updates to a user in two ways. First, we
provide evaluation metrics for a notion of compatibility to prior model
versions, specifically for generative tasks but also applicable for
discriminative tasks. We observe regression and inconsistencies between
different model versions on a diverse set of tasks and model updates. Second,
we propose a training strategy to minimize the number of inconsistencies in
model updates, involving training of a compatibility model that can enhance
task fine-tuned language models. We reduce negative flips -- instances where a
prior model version was correct, but a new model incorrect -- by up to 40% from
Llama 1 to Llama 2.
