  With increasing reliance on medical imaging in clinical practices, automated
report generation from medical images is in great demand. Existing report
generation methods typically adopt an encoder-decoder deep learning framework
to build a uni-directional image-to-report mapping. However, such a framework
ignores the bi-directional mutual associations between images and reports, thus
incurring difficulties in associating the intrinsic medical meanings between
them. Recent generative representation learning methods have demonstrated the
benefits of dual-modal learning from both image and text modalities. However,
these methods exhibit two major drawbacks for medical report generation: 1)
they tend to capture morphological information and have difficulties in
capturing subtle pathological semantic information, and 2) they predict masked
text rely on both unmasked images and text, inevitably degrading performance
when inference is based solely on images. In this study, we propose a new
report generation framework with dual-modal dynamic traceback learning (DTrace)
to overcome the two identified drawbacks and enable dual-modal learning for
medical report generation. To achieve this, our DTrace introduces a traceback
mechanism to control the semantic validity of generated content via
self-assessment. Further, our DTrace introduces a dynamic learning strategy to
adapt to various proportions of image and text input, enabling report
generation without reliance on textual input during inference. Extensive
experiments on two well-benchmarked datasets (IU-Xray and MIMIC-CXR) show that
our DTrace outperforms state-of-the-art medical report generation methods.
