  A well-formulated benchmark plays a critical role in spurring advancements in
the natural language processing (NLP) field, as it allows objective and precise
evaluation of diverse models. As modern language models (LMs) have become more
elaborate and sophisticated, more difficult benchmarks that require linguistic
knowledge and reasoning have been proposed. However, most of these benchmarks
only support English, and great effort is necessary to construct benchmarks for
other low resource languages. To this end, we propose a new benchmark named
Korean balanced evaluation of significant tasks (KoBEST), which consists of
five Korean-language downstream tasks. Professional Korean linguists designed
the tasks that require advanced Korean linguistic knowledge. Moreover, our data
is purely annotated by humans and thoroughly reviewed to guarantee high data
quality. We also provide baseline models and human performance results. Our
dataset is available on the Huggingface.
