  Although great progress has been made by previous table understanding methods
including recent approaches based on large language models (LLMs), they rely
heavily on the premise that given tables must be converted into a certain text
sequence (such as Markdown or HTML) to serve as model input. However, it is
difficult to access such high-quality textual table representations in some
real-world scenarios, and table images are much more accessible. Therefore, how
to directly understand tables using intuitive visual information is a crucial
and urgent challenge for developing more practical applications. In this paper,
we propose a new problem, multimodal table understanding, where the model needs
to generate correct responses to various table-related requests based on the
given table image. To facilitate both the model training and evaluation, we
construct a large-scale dataset named MMTab, which covers a wide spectrum of
table images, instructions and tasks. On this basis, we develop Table-LLaVA, a
generalist tabular multimodal large language model (MLLM), which significantly
outperforms recent open-source MLLM baselines on 23 benchmarks under held-in
and held-out settings. The code and data is available at this
https://github.com/SpursGoZmy/Table-LLaVA
