  Pipeline parallelism (PP) when training neural networks enables larger models
to be partitioned spatially, leading to both lower network communication and
overall higher hardware utilization. Unfortunately, to preserve the statistical
efficiency of sequential training, existing PP techniques sacrifice hardware
efficiency by decreasing pipeline utilization or incurring extra memory costs.
In this paper, we investigate to what extent these sacrifices are necessary. We
devise PipeMare, a simple yet robust training method that tolerates
asynchronous updates during PP execution without sacrificing utilization or
memory, which allows efficient use of fine-grained pipeline parallelism.
Concretely, when tested on ResNet and Transformer networks, asynchrony enables
PipeMare to use up to $2.7\times$ less memory or get $4.3\times$ higher
pipeline utilization, with similar model quality, when compared to
state-of-the-art synchronous PP training techniques.
