  Charts are commonly used for exploring data and communicating insights.
Generating natural language summaries from charts can be very helpful for
people in inferring key insights that would otherwise require a lot of
cognitive and perceptual efforts. We present Chart-to-text, a large-scale
benchmark with two datasets and a total of 44,096 charts covering a wide range
of topics and chart types. We explain the dataset construction process and
analyze the datasets. We also introduce a number of state-of-the-art neural
models as baselines that utilize image captioning and data-to-text generation
techniques to tackle two problem variations: one assumes the underlying data
table of the chart is available while the other needs to extract data from
chart images. Our analysis with automatic and human evaluation shows that while
our best models usually generate fluent summaries and yield reasonable BLEU
scores, they also suffer from hallucinations and factual errors as well as
difficulties in correctly explaining complex patterns and trends in charts.
