  When pneumonia is not found on a chest X-ray, should the report describe this
negative observation or omit it? We argue that this question cannot be answered
from the X-ray alone and requires a pragmatic perspective, which captures the
communicative goal that radiology reports serve between radiologists and
patients. However, the standard image-to-text formulation for radiology report
generation fails to incorporate such pragmatic intents. Following this
pragmatic perspective, we demonstrate that the indication, which describes why
a patient comes for an X-ray, drives the mentions of negative observations and
introduce indications as additional input to report generation. With respect to
the output, we develop a framework to identify uninferable information from the
image as a source of model hallucinations, and limit them by cleaning
groundtruth reports. Finally, we use indications and cleaned groundtruth
reports to develop pragmatic models, and show that they outperform existing
methods not only in new pragmatics-inspired metrics (+4.3 Negative F1) but also
in standard metrics (+6.3 Positive F1 and +11.0 BLEU-2).
