  In the rapidly evolving field of artificial intelligence, large language
models (LLMs) have emerged as powerful tools for a myriad of applications, from
natural language processing to decision-making support systems. However, as
these models become increasingly integrated into societal frameworks, the
imperative to ensure they operate within ethical and moral boundaries has never
been more critical. This paper introduces a novel benchmark designed to measure
and compare the moral reasoning capabilities of LLMs. We present the first
comprehensive dataset specifically curated to probe the moral dimensions of LLM
outputs, addressing a wide range of ethical dilemmas and scenarios reflective
of real-world complexities.
  The main contribution of this work lies in the development of benchmark
datasets and metrics for assessing the moral identity of LLMs, which accounts
for nuance, contextual sensitivity, and alignment with human ethical standards.
Our methodology involves a multi-faceted approach, combining quantitative
analysis with qualitative insights from ethics scholars to ensure a thorough
evaluation of model performance. By applying our benchmark across several
leading LLMs, we uncover significant variations in moral reasoning capabilities
of different models. These findings highlight the importance of considering
moral reasoning in the development and evaluation of LLMs, as well as the need
for ongoing research to address the biases and limitations uncovered in our
study. We publicly release the benchmark at
https://drive.google.com/drive/u/0/folders/1k93YZJserYc2CkqP8d4B3M3sgd3kA8W7
and also open-source the code of the project at
https://github.com/agiresearch/MoralBench.
