  Reducing the requirement for densely annotated masks in medical image
segmentation is important due to cost constraints. In this paper, we consider
the problem of inferring pixel-level predictions of brain lesions by only using
image-level labels for training. By leveraging recent advances in generative
diffusion probabilistic models (DPM), we synthesize counterfactuals of "How
would a patient appear if X pathology was not present?". The difference image
between the observed patient state and the healthy counterfactual can be used
for inferring the location of pathology. We generate counterfactuals that
correspond to the minimal change of the input such that it is transformed to
healthy domain. This requires training with healthy and unhealthy data in DPMs.
We improve on previous counterfactual DPMs by manipulating the generation
process with implicit guidance along with attention conditioning instead of
using classifiers. Code is available at https://github.com/vios-s/Diff-SCM.
