  Mitigating bias in machine learning models is a critical endeavor for
ensuring fairness and equity. In this paper, we propose a novel approach to
address bias by leveraging pixel image attributions to identify and regularize
regions of images containing significant information about bias attributes. Our
method utilizes a model-agnostic approach to extract pixel attributions by
employing a convolutional neural network (CNN) classifier trained on small
image patches. By training the classifier to predict a property of the entire
image using only a single patch, we achieve region-based attributions that
provide insights into the distribution of important information across the
image. We propose utilizing these attributions to introduce targeted noise into
datasets with confounding attributes that bias the data, thereby constraining
neural networks from learning these biases and emphasizing the primary
attributes. Our approach demonstrates its efficacy in enabling the training of
unbiased classifiers on heavily biased datasets.
