  Evidence data for automated fact-checking (AFC) can be in multiple modalities
such as text, tables, images, audio, or video. While there is increasing
interest in using images for AFC, previous works mostly focus on detecting
manipulated or fake images. We propose a novel task, chart-based fact-checking,
and introduce ChartBERT as the first model for AFC against chart evidence.
ChartBERT leverages textual, structural and visual information of charts to
determine the veracity of textual claims. For evaluation, we create ChartFC, a
new dataset of 15, 886 charts. We systematically evaluate 75 different
vision-language (VL) baselines and show that ChartBERT outperforms VL models,
achieving 63.8% accuracy. Our results suggest that the task is complex yet
feasible, with many challenges ahead.
