  The Large Language models (LLMs) have demonstrated supreme capabilities in
text understanding and generation, but cannot be directly applied to
cross-modal tasks without fine-tuning. This paper proposes a cross-modal
in-context learning approach, empowering the frozen LLMs to achieve multiple
audio tasks in a few-shot style without any parameter update. Specifically, we
propose a novel and LLMs-driven audio codec model, LLM-Codec, to transfer the
audio modality into the textual space, \textit{i.e.} representing audio tokens
with words or sub-words in the vocabulary of LLMs, while keeping high audio
reconstruction quality. The key idea is to reduce the modality heterogeneity
between text and audio by compressing the audio modality into a well-trained
LLMs token space. Thus, the audio representation can be viewed as a new
\textit{foreign language}, and LLMs can learn the new \textit{foreign language}
with several demonstrations. In experiments, we investigate the performance of
the proposed approach across multiple audio understanding and generation tasks,
\textit{e.g.} speech emotion classification, audio classification,
text-to-speech generation, speech enhancement, etc. The experimental results
demonstrate that the LLMs equipped with the proposed LLM-Codec, named as
UniAudio 1.5, prompted by only a few examples, can achieve the expected
functions in simple scenarios. It validates the feasibility and effectiveness
of the proposed cross-modal in-context learning approach. To facilitate
research on few-shot audio task learning and multi-modal LLMs, we have
open-sourced the LLM-Codec model.
