  The developments that language models have provided in fulfilling almost all
kinds of tasks have attracted the attention of not only researchers but also
the society and have enabled them to become products. There are commercially
successful language models available. However, users may prefer open-source
language models due to cost, data privacy, or regulations. Yet, despite the
increasing number of these models, there is no comprehensive comparison of
their performance for Turkish. This study aims to fill this gap in the
literature. A comparison is made among seven selected language models based on
their contextual learning and question-answering abilities. Turkish datasets
for contextual learning and question-answering were prepared, and both
automatic and human evaluations were conducted. The results show that for
question-answering, continuing pretraining before fine-tuning with
instructional datasets is more successful in adapting multilingual models to
Turkish and that in-context learning performances do not much related to
question-answering performances.
