  In response to the urgent demand for grid stability and the complex
challenges posed by renewable energy integration and electricity market
dynamics, the power sector increasingly seeks innovative technological
solutions. In this context, large language models (LLMs) have become a key
technology to improve efficiency and promote intelligent progress in the power
sector with their excellent natural language processing, logical reasoning, and
generalization capabilities. Despite their potential, the absence of a
performance evaluation benchmark for LLM in the power sector has limited the
effective application of these technologies. Addressing this gap, our study
introduces "ElecBench", an evaluation benchmark of LLMs within the power
sector. ElecBench aims to overcome the shortcomings of existing evaluation
benchmarks by providing comprehensive coverage of sector-specific scenarios,
deepening the testing of professional knowledge, and enhancing decision-making
precision. The framework categorizes scenarios into general knowledge and
professional business, further divided into six core performance metrics:
factuality, logicality, stability, security, fairness, and expressiveness, and
is subdivided into 24 sub-metrics, offering profound insights into the
capabilities and limitations of LLM applications in the power sector. To ensure
transparency, we have made the complete test set public, evaluating the
performance of eight LLMs across various scenarios and metrics. ElecBench
aspires to serve as the standard benchmark for LLM applications in the power
sector, supporting continuous updates of scenarios, metrics, and models to
drive technological progress and application.
