  Shortcut learning is when a model -- e.g. a cardiac disease classifier --
exploits correlations between the target label and a spurious shortcut feature,
e.g. a pacemaker, to predict the target label based on the shortcut rather than
real discriminative features. This is common in medical imaging, where
treatment and clinical annotations correlate with disease labels, making them
easy shortcuts to predict disease. We propose a novel detection and
quantification of the impact of potential shortcut features via a fast
diffusion-based counterfactual image generation that can synthetically remove
or add shortcuts. Via a novel inpainting-based modification we spatially limit
the changes made with no extra inference step, encouraging the removal of
spatially constrained shortcut features while ensuring that the shortcut-free
counterfactuals preserve their remaining image features to a high degree. Using
these, we assess how shortcut features influence model predictions.
  This is enabled by our second contribution: An efficient diffusion-based
counterfactual explanation method with significant inference speed-up at
comparable image quality as state-of-the-art. We confirm this on two large
chest X-ray datasets, a skin lesion dataset, and CelebA. Our code is publicly
available at fastdime.compute.dtu.dk.
