  Medical imaging contains the essential information for rendering diagnostic
and treatment decisions. Inspecting (visual perception) and interpreting image
to generate a report are tedious clinical routines for a radiologist where
automation is expected to greatly reduce the workload. Despite rapid
development of natural image captioning, computer-aided medical image visual
perception and interpretation remain a challenging task, largely due to the
lack of high-quality annotated image-report pairs and tailor-made generative
models for sufficient extraction and exploitation of localized semantic
features, particularly those associated with abnormalities. To tackle these
challenges, we present Vispi, an automatic medical image interpretation system,
which first annotates an image via classifying and localizing common thoracic
diseases with visual support and then followed by report generation from an
attentive LSTM model. Analyzing an open IU X-ray dataset, we demonstrate a
superior performance of Vispi in disease classification, localization and
report generation using automatic performance evaluation metrics ROUGE and
CIDEr.
