  The chest X-Ray (CXR) is the one of the most common clinical exam used to
diagnose thoracic diseases and abnormalities. The volume of CXR scans generated
daily in hospitals is huge. Therefore, an automated diagnosis system able to
save the effort of doctors is of great value. At present, the applications of
artificial intelligence in CXR diagnosis usually use pattern recognition to
classify the scans. However, such methods rely on labeled databases, which are
costly and usually have large error rates. In this work, we built a database
containing more than 12,000 CXR scans and radiological reports, and developed a
model based on deep convolutional neural network and recurrent network with
attention mechanism. The model learns features from the CXR scans and the
associated raw radiological reports directly; no additional labeling of the
scans are needed. The model provides automated recognition of given scans and
generation of reports. The quality of the generated reports was evaluated with
both the CIDEr scores and by radiologists as well. The CIDEr scores are found
to be around 5.8 on average for the testing dataset. Further blind evaluation
suggested a comparable performance against human radiologist.
