  Effective adaptation to distribution shifts in training data is pivotal for
sustaining robustness in neural networks, especially when removing specific
biases or outdated information, a process known as machine unlearning.
Traditional approaches typically assume that data variations are random, which
makes it difficult to adjust the model parameters accurately to remove patterns
and characteristics from unlearned data. In this work, we present Unlearning
Information Bottleneck (UIB), a novel information-theoretic framework designed
to enhance the process of machine unlearning that effectively leverages the
influence of systematic patterns and biases for parameter adjustment. By
proposing a variational upper bound, we recalibrate the model parameters
through a dynamic prior that integrates changes in data distribution with an
affordable computational cost, allowing efficient and accurate removal of
outdated or unwanted data patterns and biases. Our experiments across various
datasets, models, and unlearning methods demonstrate that our approach
effectively removes systematic patterns and biases while maintaining the
performance of models post-unlearning.
