  Figures, such as bar charts, pie charts, and line plots, are widely used to
convey important information in a concise format. They are usually
human-friendly but difficult for computers to process automatically. In this
work, we investigate the problem of figure captioning where the goal is to
automatically generate a natural language description of the figure. While
natural image captioning has been studied extensively, figure captioning has
received relatively little attention and remains a challenging problem. First,
we introduce a new dataset for figure captioning, FigCAP, based on FigureQA.
Second, we propose two novel attention mechanisms. To achieve accurate
generation of labels in figures, we propose Label Maps Attention. To model the
relations between figure labels, we propose Relation Maps Attention. Third, we
use sequence-level training with reinforcement learning in order to directly
optimizes evaluation metrics, which alleviates the exposure bias issue and
further improves the models in generating long captions. Extensive experiments
show that the proposed method outperforms the baselines, thus demonstrating a
significant potential for the automatic captioning of vast repositories of
figures.
