  Data augmentation is widely used to mitigate data bias in the training
dataset. However, data augmentation exposes machine learning models to privacy
attacks, such as membership inference attacks. In this paper, we propose an
effective combination of data augmentation and machine unlearning, which can
reduce data bias while providing a provable defense against known attacks.
Specifically, we maintain the fairness of the trained model with
diffusion-based data augmentation, and then utilize multi-shard unlearning to
remove identifying information of original data from the ML model for
protection against privacy attacks. Experimental evaluation across diverse
datasets demonstrates that our approach can achieve significant improvements in
bias reduction as well as robustness against state-of-the-art privacy attacks.
