  Automating report generation for medical imaging promises to reduce workload
and assist diagnosis in clinical practice. Recent work has shown that deep
learning models can successfully caption natural images. However, learning from
medical data is challenging due to the diversity and uncertainty inherent in
the reports written by different radiologists with discrepant expertise and
experience. To tackle these challenges, we propose variational topic inference
for automatic report generation. Specifically, we introduce a set of topics as
latent variables to guide sentence generation by aligning image and language
modalities in a latent space. The topics are inferred in a conditional
variational inference framework, with each topic governing the generation of a
sentence in the report. Further, we adopt a visual attention module that
enables the model to attend to different locations in the image and generate
more informative descriptions. We conduct extensive experiments on two
benchmarks, namely Indiana U. Chest X-rays and MIMIC-CXR. The results
demonstrate that our proposed variational topic inference method can generate
novel reports rather than mere copies of reports used in training, while still
achieving comparable performance to state-of-the-art methods in terms of
standard language generation criteria.
