  AI accelerator processing capabilities and memory constraints largely dictate
the scale in which machine learning workloads (e.g., training and inference)
can be executed within a desirable time frame. Training a state of the art,
transformer-based model today requires use of GPU-accelerated high performance
computers with high-speed interconnects. As datasets and models continue to
increase in size, computational requirements and memory demands for AI also
continue to grow. These challenges have inspired the development of distributed
algorithm and circuit-based optimization techniques that enable the ability to
progressively scale models in multi-node environments, efficiently minimize
neural network cost functions for faster convergence, and store more parameters
into a set number of available resources. In our research project, we focus on
parallel and distributed machine learning algorithm development, specifically
for optimizing the data processing and pre-training of a set of 5
encoder-decoder LLMs, ranging from 580 million parameters to 13 billion
parameters. We performed a fine-grained study to quantify the relationships
between three ML parallelism methods, specifically exploring Microsoft
DeepSpeed Zero Redundancy Optimizer (ZeRO) stages.
