  The idea of personality in descriptive psychology, traditionally defined
through observable behavior, has now been extended to Large Language Models
(LLMs) to better understand their behavior. This raises a question: do LLMs
exhibit distinct and consistent personality traits, similar to humans? Existing
self-assessment personality tests, while applicable, lack the necessary
validity and reliability for precise personality measurements. To address this,
we introduce TRAIT, a new tool consisting of 8K multi-choice questions designed
to assess the personality of LLMs with validity and reliability. TRAIT is built
on the psychometrically validated human questionnaire, Big Five Inventory (BFI)
and Short Dark Triad (SD-3), enhanced with the ATOMIC10X knowledge graph for
testing personality in a variety of real scenarios. TRAIT overcomes the
reliability and validity issues when measuring personality of LLM with
self-assessment, showing the highest scores across three metrics: refusal rate,
prompt sensitivity, and option order sensitivity. It reveals notable insights
into personality of LLM: 1) LLMs exhibit distinct and consistent personality,
which is highly influenced by their training data (i.e., data used for
alignment tuning), and 2) current prompting techniques have limited
effectiveness in eliciting certain traits, such as high psychopathy or low
conscientiousness, suggesting the need for further research in this direction.
