  Symbolic Music, akin to language, can be encoded in discrete symbols. Recent
research has extended the application of large language models (LLMs) such as
GPT-4 and Llama2 to the symbolic music domain including understanding and
generation. Yet scant research explores the details of how these LLMs perform
on advanced music understanding and conditioned generation, especially from the
multi-step reasoning perspective, which is a critical aspect in the
conditioned, editable, and interactive human-computer co-creation process. This
study conducts a thorough investigation of LLMs' capability and limitations in
symbolic music processing. We identify that current LLMs exhibit poor
performance in song-level multi-step music reasoning, and typically fail to
leverage learned music knowledge when addressing complex musical tasks. An
analysis of LLMs' responses highlights distinctly their pros and cons. Our
findings suggest achieving advanced musical capability is not intrinsically
obtained by LLMs, and future research should focus more on bridging the gap
between music knowledge and reasoning, to improve the co-creation experience
for musicians.
