  Recent advancements in Large Language Models (LLMs) have significantly
enhanced the capabilities of conversational agents, making them applicable to
various fields (e.g., education). Despite their progress, the evaluation of the
agents often overlooks the complexities of real-world conversations, such as
real-time interactions, multi-party dialogues, and extended contextual
dependencies. To bridge this gap, we introduce DialSim, a real-time dialogue
simulator. In this simulator, an agent is assigned the role of a character from
popular TV shows, requiring it to respond to spontaneous questions using past
dialogue information and to distinguish between known and unknown information.
Key features of DialSim include evaluating the agent's ability to respond
within a reasonable time limit, handling long-term multi-party dialogues, and
managing adversarial settings (e.g., swap character names) to challenge the
agent's reliance on pre-trained knowledge. We utilized this simulator to
evaluate the latest conversational agents and analyze their limitations. Our
experiments highlight both the strengths and weaknesses of these agents,
providing valuable insights for future improvements in the field of
conversational AI. DialSim is available at
https://github.com/jiho283/Simulator.
