  Large language models (LLMs) with long sequences begin to power more and more
fundamentally new applications we use every day. Existing methods for
long-sequence LLM training are neither efficient nor compatible with
commonly-used training algorithms such as FlashAttention. We design InternEvo
to address these issues. InternEvo decouples all of the sharding dimensions
into a new hierarchical space, and systematically analyzes the memory and
communication cost of LLM training. Then, it generates an effective hybrid
parallelism strategy. We design a new selective overlap mechanism to mitigate
the communication overhead introduced by the hybrid parallelism. We also
implement memory management techniques to reduce GPU memory fragmentation.
Evaluation results show that InternEvo generates parallelization strategies
that match or outperform existing methods in model FLOPs utilization.
