  Speech encoders pretrained through self-supervised learning (SSL) have
demonstrated remarkable performance in various downstream tasks, including
Spoken Language Understanding (SLU) and Automatic Speech Recognition (ASR). For
instance, fine-tuning SSL models for such tasks has shown significant
potential, leading to improvements in the SOTA performance across challenging
datasets. In contrast to existing research, this paper contributes by comparing
the effectiveness of SSL approaches in the context of (i) the low-resource
spoken Tunisian Arabic dialect and (ii) its combination with a low-resource SLU
and ASR scenario, where only a few semantic annotations are available for
fine-tuning. We conduct experiments using many SSL speech encoders on the
TARIC-SLU dataset. We use speech encoders that were pre-trained on either
monolingual or multilingual speech data. Some of them have also been refined
without in-domain nor Tunisian data through multimodal supervised
teacher-student paradigm. This study yields numerous significant findings that
we are discussing in this paper.
