  The recent success of large language models (LLMs) has attracted widespread
interest to develop role-playing conversational agents personalized to the
characteristics and styles of different speakers to enhance their abilities to
perform both general and special purpose dialogue tasks. However, the ability
to personalize the generated utterances to speakers, whether conducted by human
or LLM, has not been well studied. To bridge this gap, our study introduces a
novel evaluation challenge: speaker verification in agent-generated
conversations, which aimed to verify whether two sets of utterances originate
from the same speaker. To this end, we assemble a large dataset collection
encompassing thousands of speakers and their utterances. We also develop and
evaluate speaker verification models under experiment setups. We further
utilize the speaker verification models to evaluate the personalization
abilities of LLM-based role-playing models. Comprehensive experiments suggest
that the current role-playing models fail in accurately mimicking speakers,
primarily due to their inherent linguistic characteristics.
