  Predictive machine learning models are becoming increasingly deployed in
high-stakes contexts involving sensitive personal data; in these contexts,
there is a trade-off between model explainability and data privacy. In this
work, we push the boundaries of this trade-off: with a focus on foundation
models for image classification fine-tuning, we reveal unforeseen privacy risks
of post-hoc model explanations and subsequently offer mitigation strategies for
such risks. First, we construct VAR-LRT and L1/L2-LRT, two new membership
inference attacks based on feature attribution explanations that are
significantly more successful than existing explanation-leveraging attacks,
particularly in the low false-positive rate regime that allows an adversary to
identify specific training set members with confidence. Second, we find
empirically that optimized differentially private fine-tuning substantially
diminishes the success of the aforementioned attacks, while maintaining high
model accuracy. We carry out a systematic empirical investigation of our 2 new
attacks with 5 vision transformer architectures, 5 benchmark datasets, 4
state-of-the-art post-hoc explanation methods, and 4 privacy strength settings.
