  Zero-shot spoken language understanding (SLU) enables systems to comprehend
user utterances in new domains without prior exposure to training data. Recent
studies often rely on large language models (LLMs), leading to excessive
footprints and complexity. This paper proposes the use of Whisper, a standalone
speech processing model, for zero-shot end-to-end (E2E) SLU. To handle unseen
semantic labels, SLU tasks are integrated into a question-answering (QA)
framework, which prompts the Whisper decoder for semantics deduction. The
system is efficiently trained with prefix-tuning, optimising a minimal set of
parameters rather than the entire Whisper model. We show that the proposed
system achieves a 40.7% absolute gain for slot filling (SLU-F1) on SLURP
compared to a recently introduced zero-shot benchmark. Furthermore, it performs
comparably to a Whisper-GPT-2 modular system under both in-corpus and
cross-corpus evaluation settings, but with a relative 34.8% reduction in model
parameters.
