  We investigate the effect of different model architectures, training
objectives, hyperparameter settings and decoding procedures on the diversity of
automatically generated image captions. Our results show that 1) simple
decoding by naive sampling, coupled with low temperature is a competitive and
fast method to produce diverse and accurate caption sets; 2) training with
CIDEr-based reward using Reinforcement learning harms the diversity properties
of the resulting generator, which cannot be mitigated by manipulating decoding
parameters. In addition, we propose a new metric AllSPICE for evaluating both
accuracy and diversity of a set of captions by a single value.
