  Automated radiographic report generation is a challenging cross-domain task
that aims to automatically generate accurate and semantic-coherence reports to
describe medical images. Despite the recent progress in this field, there are
still many challenges at least in the following aspects. First, radiographic
images are very similar to each other, and thus it is difficult to capture the
fine-grained visual differences using CNN as the visual feature extractor like
many existing methods. Further, semantic information has been widely applied to
boost the performance of generation tasks (e.g. image captioning), but existing
methods often fail to provide effective medical semantic features. Toward
solving those problems, in this paper, we propose a memory-augmented sparse
attention block utilizing bilinear pooling to capture the higher-order
interactions between the input fine-grained image features while producing
sparse attention. Moreover, we introduce a novel Medical Concepts Generation
Network (MCGN) to predict fine-grained semantic concepts and incorporate them
into the report generation process as guidance. Our proposed method shows
promising performance on the recently released largest benchmark MIMIC-CXR. It
outperforms multiple state-of-the-art methods in image captioning and medical
report generation.
