  Large language models (LLMs) have revolutionized the field of natural
language processing, extending their strong capabilities into multi-modal
domains. Thus, it is vital to define proper and diversified metrics for the
evaluation of LLMs.
  In this paper, we introduce matrix entropy, a novel metric rooted in
information theory and geometry principles to quantify the data compression
proficiency in LLMs. It reflects the model's ability to extract relevant
information and eliminate unnecessary elements, thereby providing insight into
the language model's intrinsic capability. Specifically, we demonstrate its
applicability in both single-modal (language) and multi-modal settings. For
language models, our findings reveal that the matrix entropy of representations
follows a scaling law type reduction when the model scales up, serving as a
complement to the traditional loss scaling law. For the multi-modal setting, we
also propose an evaluation method based on matrix entropy for assessing
alignment quality and we find that modern large multi-modal models exhibit
great alignment performance.
