  Due to the common content of anatomy, radiology images with their
corresponding reports exhibit high similarity. Such inherent data bias can
predispose automatic report generation models to learn entangled and spurious
representations resulting in misdiagnostic reports. To tackle these, we propose
a novel \textbf{Co}unter\textbf{F}actual \textbf{E}xplanations-based framework
(CoFE) for radiology report generation. Counterfactual explanations serve as a
potent tool for understanding how decisions made by algorithms can be changed
by asking ``what if'' scenarios. By leveraging this concept, CoFE can learn
non-spurious visual representations by contrasting the representations between
factual and counterfactual images. Specifically, we derive counterfactual
images by swapping a patch between positive and negative samples until a
predicted diagnosis shift occurs. Here, positive and negative samples are the
most semantically similar but have different diagnosis labels. Additionally,
CoFE employs a learnable prompt to efficiently fine-tune the pre-trained large
language model, encapsulating both factual and counterfactual content to
provide a more generalizable prompt representation. Extensive experiments on
two benchmarks demonstrate that leveraging the counterfactual explanations
enables CoFE to generate semantically coherent and factually complete reports
and outperform in terms of language generation and clinical efficacy metrics.
