  Deep learning is experiencing a rise in foundation models that are expected
to lead in various fields. The massive number of parameters necessitates the
use of tensor model parallelism (TMP) in foundation model training. However,
TMP requires frequent communication operations which significantly reduces the
training efficiency. In this paper, we present Oases, an automated TMP method
with overlapped communication to accelerate foundation model training. Oases
proposes a fine-grained training schedule to maximize overlapping communication
and computation operations that have data dependence. Additionally, we design
the Oases planner that searches for the best model parallel strategy to achieve
further accelerations. Unlike existing methods, Oases planner is specifically
tailored to model the cost of overlapped communication-computation operations.
We evaluate Oases on various model settings and train environments, and compare
Oases to four stat-of-the-art implementations. Experimental results demonstrate
that Oases achieves speedups of 1.01--1.48X over the fastest baseline, and
speedups of up to 1.9X over Megatron-LM.
