  Chest X-Ray (CXR) images are commonly used for clinical screening and
diagnosis. Automatically writing reports for these images can considerably
lighten the workload of radiologists for summarizing descriptive findings and
conclusive impressions. The complex structures between and within sections of
the reports pose a great challenge to the automatic report generation.
Specifically, the section Impression is a diagnostic summarization over the
section Findings; and the appearance of normality dominates each section over
that of abnormality. Existing studies rarely explore and consider this
fundamental structure information. In this work, we propose a novel framework
that exploits the structure information between and within report sections for
generating CXR imaging reports. First, we propose a two-stage strategy that
explicitly models the relationship between Findings and Impression. Second, we
design a novel cooperative multi-agent system that implicitly captures the
imbalanced distribution between abnormality and normality. Experiments on two
CXR report datasets show that our method achieves state-of-the-art performance
in terms of various evaluation metrics. Our results expose that the proposed
approach is able to generate high-quality medical reports through integrating
the structure information.
