  This paper explores the instruction fine-tuning technique for
speech-to-semantic tasks by introducing a unified end-to-end (E2E) framework
that generates target text conditioned on a task-related prompt for audio data.
We pre-train the model using large and diverse data, where instruction-speech
pairs are constructed via a text-to-speech (TTS) system. Extensive experiments
demonstrate that our proposed model achieves state-of-the-art (SOTA) results on
many benchmarks covering speech named entity recognition, speech sentiment
analysis, speech question answering, and more, after fine-tuning. Furthermore,
the proposed model achieves competitive performance in zero-shot and few-shot
scenarios. To facilitate future work on instruction fine-tuning for
speech-to-semantic tasks, we release our instruction dataset and code.
