  Large language models (LLMs) have gained considerable attention for
Artificial Intelligence Generated Content (AIGC), particularly with the
emergence of ChatGPT. However, the direct adaptation of continuous speech to
LLMs that process discrete tokens remains an unsolved challenge, hindering the
application of LLMs for speech generation. The advanced speech LMs are in the
corner, as that speech signals encapsulate a wealth of information, including
speaker and emotion, beyond textual data alone. Prompt tuning has demonstrated
notable gains in parameter efficiency and competitive performance on some
speech classification tasks. However, the extent to which prompts can
effectively elicit generation tasks from speech LMs remains an open question.
In this paper, we present pioneering research that explores the application of
prompt tuning to stimulate speech LMs for various generation tasks, within a
unified framework called SpeechGen, with around 10M trainable parameters. The
proposed unified framework holds great promise for efficiency and
effectiveness, particularly with the imminent arrival of advanced speech LMs,
which will significantly enhance the capabilities of the framework. The code
and demos of SpeechGen will be available on the project website:
\url{https://ga642381.github.io/SpeechPrompt/speechgen}
