  Modern Deep Learning (DL) models have grown to sizes requiring massive
clusters of specialized, high-end nodes to train. Designing such clusters to
maximize both performance and utilization--to amortize their steep cost--is a
challenging task requiring careful balance of compute, memory, and network
resources. Moreover, a plethora of each model's tuning knobs drastically affect
the performance, with optimal values often depending on the underlying
cluster's characteristics, which necessitates a complex cluster-workload
co-design process. To facilitate the design space exploration of such massive
DL training clusters, we introduce COMET, a holistic cluster design methodology
and workflow to jointly study the impact of parallelization strategies and key
cluster resource provisioning on the performance of distributed DL training. We
develop a step-by-step process to establish a reusable and flexible
methodology, and demonstrate its application with case studies of training
large models on cluster configurations of variable compute, memory, and network
resources. Our case studies demonstrate COMET's utility in identifying
promising architectural optimization directions and guiding system designers in
configuring key model and cluster parameters. To illustrate, cluster
configuration comparisons identify performance differences of up to 7.7x and
highlight performance optimization opportunities of up to 1.4x when employing
memory expansion as an optimization technique.
