  Mammogram benign or malignant classification with only image-level labels is
challenging due to the absence of lesion annotations. Motivated by the
symmetric prior that the lesions on one side of breasts rarely appear in the
corresponding areas on the other side, given a diseased image, we can explore a
counterfactual problem that how would the features have behaved if there were
no lesions in the image, so as to identify the lesion areas. We derive a new
theoretical result for counterfactual generation based on the symmetric prior.
By building a causal model that entails such a prior for bilateral images, we
obtain two optimization goals for counterfactual generation, which can be
accomplished via our newly proposed counterfactual generative network. Our
proposed model is mainly composed of Generator Adversarial Network and a
\emph{prediction feedback mechanism}, they are optimized jointly and prompt
each other. Specifically, the former can further improve the classification
performance by generating counterfactual features to calculate lesion areas. On
the other hand, the latter helps counterfactual generation by the supervision
of classification loss. The utility of our method and the effectiveness of each
module in our model can be verified by state-of-the-art performance on INBreast
and an in-house dataset and ablation studies.
