  Machine learning models often make predictions based on biased features such
as gender, race, and other social attributes, posing significant fairness
risks, especially in societal applications, such as hiring, banking, and
criminal justice. Traditional approaches to addressing this issue involve
retraining or fine-tuning neural networks with fairness-aware optimization
objectives. However, these methods can be impractical due to significant
computational resources, complex industrial tests, and the associated CO2
footprint. Additionally, regular users often fail to fine-tune models because
they lack access to model parameters In this paper, we introduce the
Inference-Time Rule Eraser (Eraser), a novel method designed to address
fairness concerns by removing biased decision-making rules from deployed models
during inference without altering model weights. We begin by establishing a
theoretical foundation for modifying model outputs to eliminate biased rules
through Bayesian analysis. Next, we present a specific implementation of Eraser
that involves two stages: (1) distilling the biased rules from the deployed
model into an additional patch model, and (2) removing these biased rules from
the output of the deployed model during inference. Extensive experiments
validate the effectiveness of our approach, showcasing its superior performance
in addressing fairness concerns in AI systems.
