  Pipeline parallelism has been widely explored, but most existing schedules
lack a systematic methodology. In this paper, we propose a framework to
decompose pipeline schedules as repeating a building block and we show that the
lifespan of the building block decides the peak activation memory of the
pipeline schedule. Guided by the observations, we find that almost all existing
pipeline schedules, to the best of our knowledge, are memory inefficient. To
address this, we introduce a family of memory efficient building blocks with
controllable activation memory, which can reduce the peak activation memory to
1/2 of 1F1B without sacrificing efficiency, and even to 1/3 with comparable
throughput. We can also achieve almost zero pipeline bubbles while maintaining
the same activation memory as 1F1B. Our evaluations demonstrate that in pure
pipeline parallelism settings, our methods outperform 1F1B by from 7% to 55% in
terms of throughput. When employing a grid search over hybrid parallelism
hyperparameters in practical scenarios, our proposed methods demonstrate a 16%
throughput improvement over the 1F1B baseline for large language models.
