  Evaluating the alignment capabilities of large Vision-Language Models (VLMs)
is essential for determining their effectiveness as helpful assistants.
However, existing benchmarks primarily focus on basic abilities using nonverbal
methods, such as yes-no and multiple-choice questions. In this paper, we
address this gap by introducing AlignMMBench, a comprehensive alignment
benchmark specifically designed for emerging Chinese VLMs. This benchmark is
meticulously curated from real-world scenarios and Chinese Internet sources,
encompassing thirteen specific tasks across three categories, and includes both
single-turn and multi-turn dialogue scenarios. Incorporating a prompt rewrite
strategy, AlignMMBench encompasses 1,054 images and 4,978 question-answer
pairs. To facilitate the evaluation pipeline, we propose CritiqueVLM, a
rule-calibrated evaluator that exceeds GPT-4's evaluation ability. Finally, we
report the performance of representative VLMs on AlignMMBench, offering
insights into the capabilities and limitations of different VLM architectures.
All evaluation codes and data are available on https://alignmmbench.github.io.
