  Deep learning (DL) has become a key component of modern software. In the "big
model" era, the rich features of DL-based software substantially rely on
powerful DL models, e.g., BERT, GPT-3, and the recently emerging GPT-4, which
are trained on the powerful cloud with large datasets. Hence, training
effective DL models has become a vital stage in the whole software lifecycle.
When training deep learning models, especially those big models, developers
need to parallelize and distribute the computation and memory resources amongst
multiple devices in the training process, which is known as distributed deep
learning training, or distributed training for short. However, the unique
challenges that developers encounter in distributed training process have not
been studied in the software engineering community. Given the increasingly
heavy dependence of current DL-based software on distributed training, this
paper aims to fill in the knowledge gap and presents the first comprehensive
study on developers' issues in distributed training. To this end, we analyze
1,131 real-world developers' issues about using these frameworks reported on
Stack Overflow and GitHub. We construct a fine-grained taxonomy consisting of
30 categories regarding the fault symptoms and summarize common fix patterns
for different symptoms. Based on the results, we suggest actionable
implications on research avenues that can potentially facilitate the
distributed training to develop DL-based software, such as focusing on the
frequent and common fix patterns when designing testing or debugging tools,
developing efficient testing and debugging techniques for communication
configuration along with the synthesis of network configuration analysis,
designing new multi-device checkpoint-and-replay techniques to help
reproduction, and designing serverless APIs for cloud platforms.
