  Generating radiology reports automatically reduces the workload of
radiologists and helps the diagnoses of specific diseases. Many existing
methods take this task as modality transfer process. However, since the key
information related to disease accounts for a small proportion in both image
and report, it is hard for the model to learn the latent relation between the
radiology image and its report, thus failing to generate fluent and accurate
radiology reports. To tackle this problem, we propose a memory-based
cross-modal semantic alignment model (MCSAM) following an encoder-decoder
paradigm. MCSAM includes a well initialized long-term clinical memory bank to
learn disease-related representations as well as prior knowledge for different
modalities to retrieve and use the retrieved memory to perform feature
consolidation. To ensure the semantic consistency of the retrieved cross modal
prior knowledge, a cross-modal semantic alignment module (SAM) is proposed. SAM
is also able to generate semantic visual feature embeddings which can be added
to the decoder and benefits report generation. More importantly, to memorize
the state and additional information while generating reports with the decoder,
we use learnable memory tokens which can be seen as prompts. Extensive
experiments demonstrate the promising performance of our proposed method which
generates state-of-the-art performance on the MIMIC-CXR dataset.
