  Causal generative modelling is gaining interest in medical imaging due to its
ability to answer interventional and counterfactual queries. Most work focuses
on generating counterfactual images that look plausible, using auxiliary
classifiers to enforce effectiveness of simulated interventions. We investigate
pitfalls in this approach, discovering the issue of attribute amplification,
where unrelated attributes are spuriously affected during interventions,
leading to biases across protected characteristics and disease status. We show
that attribute amplification is caused by the use of hard labels in the
counterfactual training process and propose soft counterfactual fine-tuning to
mitigate this issue. Our method substantially reduces the amplification effect
while maintaining effectiveness of generated images, demonstrated on a large
chest X-ray dataset. Our work makes an important advancement towards more
faithful and unbiased causal modelling in medical imaging.
