  Wav2Prompt is proposed which allows straightforward integration between
spoken input and a text-based large language model (LLM). Wav2Prompt uses a
simple training process with only the same data used to train an automatic
speech recognition (ASR) model. After training, Wav2Prompt learns continuous
representations from speech and uses them as LLM prompts. To avoid task
over-fitting issues found in prior work and preserve the emergent abilities of
LLMs, Wav2Prompt takes LLM token embeddings as the training targets and
utilises a continuous integrate-and-fire mechanism for explicit speech-text
alignment. Therefore, a Wav2Prompt-LLM combination can be applied to zero-shot
spoken language tasks such as speech translation (ST), speech understanding
(SLU), speech question answering (SQA) and spoken-query-based QA (SQQA). It is
shown that for these zero-shot tasks, Wav2Prompt performs similarly to an
ASR-LLM cascade and better than recent prior work. If relatively small amounts
of task-specific paired data are available in few-shot scenarios, the
Wav2Prompt-LLM combination can be end-to-end (E2E) fine-tuned. The
Wav2Prompt-LLM combination then yields greatly improved results relative to an
ASR-LLM cascade for the above tasks. For instance, for English-French ST with
the BLOOMZ-7B1 LLM, a Wav2Prompt-LLM combination gave a 8.5 BLEU point increase
over an ASR-LLM cascade.
