  While recent advancements in speech language models have achieved significant
progress, they face remarkable challenges in modeling the long acoustic
sequences of neural audio codecs. In this paper, we introduce
\textbf{G}enerative \textbf{P}re-trained \textbf{S}peech \textbf{T}ransformer
(GPST), a hierarchical transformer designed for efficient speech language
modeling. GPST quantizes audio waveforms into two distinct types of discrete
speech representations and integrates them within a hierarchical transformer
architecture, allowing for a unified one-stage generation process and enhancing
Hi-Res audio generation capabilities. By training on large corpora of speeches
in an end-to-end unsupervised manner, GPST can generate syntactically
consistent speech with diverse speaker identities. Given a brief 3-second
prompt, GPST can produce natural and coherent personalized speech,
demonstrating in-context learning abilities. Moreover, our approach can be
easily extended to spoken cross-lingual speech generation by incorporating
multi-lingual semantic tokens and universal acoustic tokens. Experimental
results indicate that GPST significantly outperforms the existing speech
language models in terms of word error rate, speech quality, and speaker
similarity. See \url{https://youngsheen.github.io/GPST/demo} for demo samples.
