  We present TMMLU+, a new benchmark designed for Traditional Chinese language
understanding. TMMLU+ is a multi-choice question-answering dataset with 66
subjects from elementary to professional level. It is six times larger and
boasts a more balanced subject distribution than its predecessor, Taiwan
Massive Multitask Language Understanding (TMMLU). We also benchmark
closed-source models and 26 open-weight Chinese large language models (LLMs) of
parameters ranging from 1.8B to 72B on the proposed TMMLU+. Our findings reveal
that (1.) Traditional Chinese models still trail behind their Simplified
Chinese counterparts, highlighting a need for more focused advancements in LLMs
catering to Traditional Chinese. (2.) Current LLMs still fall short of human
performance in average scores, indicating a potential need for future research
to delve deeper into social science and humanities subjects. (3.) Among all the
tokenization compression metrics examined, we identify that only the fertility
score uniquely demonstrates strong correlations with our benchmark results. We
foresee that TMMLU+ will pinpoint areas for future model improvement, thereby
narrowing the gap between machine and human linguistic capabilities and
supporting researchers in developing Traditional Chinese LLMs. Our dataset,
along with the benchmark source code, is accessible at
huggingface.co/datasets/ikala/tmmluplus.
