  Large Language Models (LLMs) are transforming diverse fields and gaining
increasing influence as human proxies. This development underscores the urgent
need for evaluating value orientations and understanding of LLMs to ensure
their responsible integration into public-facing applications. This work
introduces ValueBench, the first comprehensive psychometric benchmark for
evaluating value orientations and value understanding in LLMs. ValueBench
collects data from 44 established psychometric inventories, encompassing 453
multifaceted value dimensions. We propose an evaluation pipeline grounded in
realistic human-AI interactions to probe value orientations, along with novel
tasks for evaluating value understanding in an open-ended value space. With
extensive experiments conducted on six representative LLMs, we unveil their
shared and distinctive value orientations and exhibit their ability to
approximate expert conclusions in value-related extraction and generation
tasks. ValueBench is openly accessible at
https://github.com/Value4AI/ValueBench.
