  Creating open-domain chatbots requires large amounts of conversational data
and related benchmark tasks to evaluate them. Standardized evaluation tasks are
crucial for creating automatic evaluation metrics for model development;
otherwise, comparing the models would require resource-expensive human
evaluation. While chatbot challenges have recently managed to provide a
plethora of such resources for English, resources in other languages are not
yet available. In this work, we provide a starting point for Finnish
open-domain chatbot research. We describe our collection efforts to create the
Finnish chat conversation corpus FinChat, which is made available publicly.
FinChat includes unscripted conversations on seven topics from people of
different ages. Using this corpus, we also construct a retrieval-based
evaluation task for Finnish chatbot development. We observe that off-the-shelf
chatbot models trained on conversational corpora do not perform better than
chance at choosing the right answer based on automatic metrics, while humans
can do the same task almost perfectly. Similarly, in a human evaluation,
responses to questions from the evaluation set generated by the chatbots are
predominantly marked as incoherent. Thus, FinChat provides a challenging
evaluation set, meant to encourage chatbot development in Finnish.
