  Autoregressive decoding strategy is a commonly used method for text
generation tasks with pre-trained language models, while early-exiting is an
effective approach to speedup the inference stage. In this work, we propose a
novel decoding strategy named Hierarchical Skip Decoding (HSD) for efficient
autoregressive text generation. Different from existing methods that require
additional trainable components, HSD is a plug-and-play method applicable to
autoregressive text generation models, it adaptively skips decoding layers in a
hierarchical manner based on the current sequence length, thereby reducing
computational workload and allocating computation resources. Comprehensive
experiments on five text generation datasets with pre-trained language models
demonstrate HSD's advantages in balancing efficiency and text quality. With
almost half of the layers skipped, HSD can sustain 90% of the text quality
compared to vanilla autoregressive decoding, outperforming the competitive
approaches.
