  Discriminativeness is a desirable feature of image captions: captions should
describe the characteristic details of input images. However, recent
high-performing captioning models, which are trained with reinforcement
learning (RL), tend to generate overly generic captions despite their high
performance in various other criteria. First, we investigate the cause of the
unexpectedly low discriminativeness and show that RL has a deeply rooted side
effect of limiting the output words to high-frequency words. The limited
vocabulary is a severe bottleneck for discriminativeness as it is difficult for
a model to describe the details beyond its vocabulary. Then, based on this
identification of the bottleneck, we drastically recast discriminative image
captioning as a much simpler task of encouraging low-frequency word generation.
Hinted by long-tail classification and debiasing methods, we propose methods
that easily switch off-the-shelf RL models to discriminativeness-aware models
with only a single-epoch fine-tuning on the part of the parameters. Extensive
experiments demonstrate that our methods significantly enhance the
discriminativeness of off-the-shelf RL models and even outperform previous
discriminativeness-aware methods with much smaller computational costs.
Detailed analysis and human evaluation also verify that our methods boost the
discriminativeness without sacrificing the overall quality of captions.
