  Large-scale language models have become increasingly challenging and
expensive to train. Among various methods addressing this issue, Pipeline
Parallelism has been widely employed to accommodate massive model weights
within limited GPU memory. This paper introduces Hanayo, a wave-like pipeline
parallelism strategy that boasts a concise structure and practical
applicability, alongside a high-performance pipeline execution runtime to
tackle the challenges of pipeline strategy implementation. Hanayo mitigates the
issues of pipeline bubbles and excessive memory consumption prevalent in
existing schemes, without resorting to model duplicates as in Chimera. Our
evaluation, conducted on four distinct computing clusters and involving both
GPT-like and BERT-like architectures with up to 32 GPUs, demonstrates up to a
30.4 \% increase in throughput compared to the state-of-the-art approach.
