  In this study, we measure the moral reasoning ability of LLMs using the
Defining Issues Test - a psychometric instrument developed for measuring the
moral development stage of a person according to the Kohlberg's Cognitive Moral
Development Model. DIT uses moral dilemmas followed by a set of ethical
considerations that the respondent has to judge for importance in resolving the
dilemma, and then rank-order them by importance. A moral development stage
score of the respondent is then computed based on the relevance rating and
ranking.
  Our study shows that early LLMs such as GPT-3 exhibit a moral reasoning
ability no better than that of a random baseline, while ChatGPT, Llama2-Chat,
PaLM-2 and GPT-4 show significantly better performance on this task, comparable
to adult humans. GPT-4, in fact, has the highest post-conventional moral
reasoning score, equivalent to that of typical graduate school students.
However, we also observe that the models do not perform consistently across all
dilemmas, pointing to important gaps in their understanding and reasoning
abilities.
