  This paper explores the performance of encoder and decoder language models on
multilingual Natural Language Understanding (NLU) tasks, with a broad focus on
Germanic languages. Building upon the ScandEval benchmark, which initially was
restricted to evaluating encoder models, we extend the evaluation framework to
include decoder models. We introduce a method for evaluating decoder models on
NLU tasks and apply it to the languages Danish, Swedish, Norwegian, Icelandic,
Faroese, German, Dutch, and English. Through a series of experiments and
analyses, we address key research questions regarding the comparative
performance of encoder and decoder models, the impact of NLU task types, and
the variation across language resources. Our findings reveal that decoder
models can achieve significantly better NLU performance than encoder models,
with nuances observed across different tasks and languages. Additionally, we
investigate the correlation between decoders and task performance via a UMAP
analysis, shedding light on the unique capabilities of decoder and encoder
models. This study contributes to a deeper understanding of language model
paradigms in NLU tasks and provides valuable insights for model selection and
evaluation in multilingual settings.
