  Automatic radiology report generation is challenging as medical images or
reports are usually similar to each other due to the common content of anatomy.
This makes a model hard to capture the uniqueness of individual images and is
prone to producing undesired generic or mismatched reports. This situation
calls for learning more discriminative features that could capture even
fine-grained mismatches between images and reports. To achieve this, this paper
proposes a novel framework to learn discriminative image and report features by
distinguishing them from their closest peers, i.e., hard negatives. Especially,
to attain more discriminative features, we gradually raise the difficulty of
such a learning task by creating increasingly hard negative reports for each
image in the feature space during training, respectively. By treating the
increasingly hard negatives as auxiliary variables, we formulate this process
as a min-max alternating optimisation problem. At each iteration, conditioned
on a given set of hard negative reports, image and report features are learned
as usual by minimising the loss functions related to report generation. After
that, a new set of harder negative reports will be created by maximising a loss
reflecting image-report alignment. By solving this optimisation, we attain a
model that can generate more specific and accurate reports. It is noteworthy
that our framework enhances discriminative feature learning without introducing
extra network weights. Also, in contrast to the existing way of generating hard
negatives, our framework extends beyond the granularity of the dataset by
generating harder samples out of the training set. Experimental study on
benchmark datasets verifies the efficacy of our framework and shows that it can
serve as a plug-in to readily improve existing medical report generation
models.
