  Multi-modal large language models(MLLMs) have achieved remarkable progress
and demonstrated powerful knowledge comprehension and reasoning abilities.
However, the mastery of domain-specific knowledge, which is essential for
evaluating the intelligence of MLLMs, continues to be a challenge. Current
multi-modal benchmarks for domain-specific knowledge concentrate on
multiple-choice questions and are predominantly available in English, which
imposes limitations on the comprehensiveness of the evaluation. To this end, we
introduce CMMU, a novel benchmark for multi-modal and multi-type question
understanding and reasoning in Chinese. CMMU consists of 3,603 questions in 7
subjects, covering knowledge from primary to high school. The questions can be
categorized into 3 types: multiple-choice, multiple-response, and
fill-in-the-blank, bringing greater challenges to MLLMs. In addition, we
propose an evaluation strategy called Positional Error Variance for assessing
multiple-choice questions. The strategy aims to perform a quantitative analysis
of position bias. We evaluate seven open-source MLLMs along with GPT4-V,
Gemini-Pro, and Qwen-VL-Plus. The results demonstrate that CMMU poses a
significant challenge to the recent MLLMs. The data and code are available at
https://github.com/FlagOpen/CMMU.
