  Despite the recent advances in automatically describing image contents, their
applications have been mostly limited to image caption datasets containing
natural images (e.g., Flickr 30k, MSCOCO). In this paper, we present a deep
learning model to efficiently detect a disease from an image and annotate its
contexts (e.g., location, severity and the affected organs). We employ a
publicly available radiology dataset of chest x-rays and their reports, and use
its image annotations to mine disease names to train convolutional neural
networks (CNNs). In doing so, we adopt various regularization techniques to
circumvent the large normal-vs-diseased cases bias. Recurrent neural networks
(RNNs) are then trained to describe the contexts of a detected disease, based
on the deep CNN features. Moreover, we introduce a novel approach to use the
weights of the already trained pair of CNN/RNN on the domain-specific
image/text dataset, to infer the joint image/text contexts for composite image
labeling. Significantly improved image annotation results are demonstrated
using the recurrent neural cascade model by taking the joint image/text
contexts into account.
