  As language models are often deployed as chatbot assistants, it becomes a
virtue for models to engage in conversations in a user's first language. While
these models are trained on a wide range of languages, a comprehensive
evaluation of their proficiency in low-resource languages such as Korean has
been lacking. In this work, we introduce KoDialogBench, a benchmark designed to
assess language models' conversational capabilities in Korean. To this end, we
collect native Korean dialogues on daily topics from public sources, or
translate dialogues from other languages. We then structure these conversations
into diverse test datasets, spanning from dialogue comprehension to response
selection tasks. Leveraging the proposed benchmark, we conduct extensive
evaluations and analyses of various language models to measure a foundational
understanding of Korean dialogues. Experimental results indicate that there
exists significant room for improvement in models' conversation skills.
Furthermore, our in-depth comparisons across different language models
highlight the effectiveness of recent training techniques in enhancing
conversational proficiency. We anticipate that KoDialogBench will promote the
progress towards conversation-aware Korean language models.
