  Transformer achieves promising results on various tasks. However,
self-attention suffers from quadratic memory requirements with respect to the
sequence length. Existing work focuses on reducing time and space complexity
from an algorithm perspective. In this work, we propose sequence parallelism, a
memory-efficient parallelism method to help us break input sequence length
limitation and train with longer sequences on GPUs efficiently. Our approach is
compatible with most existing parallelisms (e.g. data parallelism, pipeline
parallelism and tensor parallelism), which means our sequence parallelism makes
4D parallelism possible. More importantly, we no longer require a single device
to hold the whole sequence. That is, with sparse attention, our sequence
parallelism enables us to train transformer with infinite long sequence.
Specifically, we split the input sequence into multiple chunks and feed each
chunk into its corresponding device (i.e. GPU). To compute the attention
output, we integrated ring-style communication with self-attention calculation
and proposed Ring Self-Attention (RSA). Experiments show that sequence
parallelism performs well when scaling with batch size and sequence length.
Compared with tensor parallelism, our approach achieved $13.7\times$ and
$3.0\times$ maximum batch size and sequence length respectively when scaling up
to 64 NVIDIA P100 GPUs. With sparse attention, sequence can handle sequence
with over 114K tokens, which is over $27\times$ longer than existing sparse
attention works holding the whole sequence on a single device.
