  The rapidly evolving multimodal Large Language Models (LLMs) urgently require
new benchmarks to uniformly evaluate their performance on understanding and
textually describing music. However, due to semantic gaps between Music
Information Retrieval (MIR) algorithms and human understanding, discrepancies
between professionals and the public, and low precision of annotations,
existing music description datasets cannot serve as benchmarks. To this end, we
present MuChin, the first open-source music description benchmark in Chinese
colloquial language, designed to evaluate the performance of multimodal LLMs in
understanding and describing music. We established the Caichong Music
Annotation Platform (CaiMAP) that employs an innovative multi-person,
multi-stage assurance method, and recruited both amateurs and professionals to
ensure the precision of annotations and alignment with popular semantics.
Utilizing this method, we built a dataset with multi-dimensional,
high-precision music annotations, the Caichong Music Dataset (CaiMD), and
carefully selected 1,000 high-quality entries to serve as the test set for
MuChin. Based on MuChin, we analyzed the discrepancies between professionals
and amateurs in terms of music description, and empirically demonstrated the
effectiveness of annotated data for fine-tuning LLMs. Ultimately, we employed
MuChin to evaluate existing music understanding models on their ability to
provide colloquial descriptions of music. All data related to the benchmark,
along with the scoring code and detailed appendices, have been open-sourced
(https://github.com/CarlWangChina/MuChin/).
