  Large Language Models (LLMs) have made unprecedented breakthroughs, yet their
increasing integration into everyday life might raise societal risks due to
generated unethical content. Despite extensive study on specific issues like
bias, the intrinsic values of LLMs remain largely unexplored from a moral
philosophy perspective. This work delves into ethical values utilizing Moral
Foundation Theory. Moving beyond conventional discriminative evaluations with
poor reliability, we propose DeNEVIL, a novel prompt generation algorithm
tailored to dynamically exploit LLMs' value vulnerabilities and elicit the
violation of ethics in a generative manner, revealing their underlying value
inclinations. On such a basis, we construct MoralPrompt, a high-quality dataset
comprising 2,397 prompts covering 500+ value principles, and then benchmark the
intrinsic values across a spectrum of LLMs. We discovered that most models are
essentially misaligned, necessitating further ethical value alignment. In
response, we develop VILMO, an in-context alignment method that substantially
enhances the value compliance of LLM outputs by learning to generate
appropriate value instructions, outperforming existing competitors. Our methods
are suitable for black-box and open-source models, offering a promising initial
step in studying the ethical values of LLMs.
