  The comic domain is rapidly advancing with the development of single-page
analysis and synthesis models. However, evaluation metrics and datasets lag
behind, often limited to small-scale or single-style test sets. We introduce a
novel benchmark, CoMix, designed to evaluate the multi-task capabilities of
models in comic analysis. Unlike existing benchmarks that focus on isolated
tasks such as object detection or text recognition, CoMix addresses a broader
range of tasks including object detection, speaker identification, character
re-identification, reading order, and multi-modal reasoning tasks like
character naming and dialogue generation. Our benchmark comprises three
existing datasets with expanded annotations to support multi-task evaluation.
To mitigate the over-representation of manga-style data, we have incorporated a
new dataset of carefully selected American comic-style books, thereby enriching
the diversity of comic styles. CoMix is designed to assess pre-trained models
in zero-shot and limited fine-tuning settings, probing their transfer
capabilities across different comic styles and tasks. The validation split of
the benchmark is publicly available for research purposes, and an evaluation
server for the held-out test split is also provided. Comparative results
between human performance and state-of-the-art models reveal a significant
performance gap, highlighting substantial opportunities for advancements in
comic understanding. The dataset, baseline models, and code are accessible at
the repository link. This initiative sets a new standard for comprehensive
comic analysis, providing the community with a common benchmark for evaluation
on a large and varied set.
