  DL frameworks are the basis of constructing all DL programs and models, and
thus their bugs could lead to the unexpected behaviors of any DL program or
model relying on them. Such a wide effect demonstrates the necessity and
importance of guaranteeing DL frameworks' quality. Understanding the
characteristics of DL framework bugs is a fundamental step for this quality
assurance task, facilitating designing effective bug detection and debugging
approaches. Hence, in this work we conduct the most large-scale study on 1,000
bugs from four popular and diverse DL frameworks (i.e., TensorFlow, PyTorch,
MXNet, and DL4J). By analyzing the root causes and symptoms of DL framework
bugs associated with 5 components decomposed from DL frameworks, as well as
measuring test coverage achieved by three state-of-the-art testing techniques,
we obtain 12 major findings for the comprehensive understanding of DL framework
bugs and the current status of existing DL framework testing practice, and then
provide a series of actionable guidelines for better DL framework bug detection
and debugging. Finally, based on the guidelines, we design and implement a
prototype DL-framework testing tool, called TenFuzz, which is evaluated to be
effective and finds 3 unknown bugs on the latest TensorFlow framework in a
preliminary study, indicating the significance of our guidelines.
