  Distributed machine learning is becoming increasingly popular for
geo-distributed data analytics, facilitating the collaborative analysis of data
scattered across data centers in different regions. This paradigm eliminates
the need for centralizing sensitive raw data in one location but faces the
significant challenge of high parameter synchronization delays, which stems
from the constraints of bandwidth-limited, heterogeneous, and fluctuating
wide-area networks. Prior research has focused on optimizing the
synchronization topology, evolving from starlike to tree-based structures.
However, these solutions typically depend on regular tree structures and lack
an adequate topology metric, resulting in limited improvements. This paper
proposes NetStorm, an adaptive and highly efficient communication scheduler
designed to speed up parameter synchronization across geo-distributed data
centers. First, it establishes an effective metric for optimizing a multi-root
FAPT synchronization topology. Second, a network awareness module is developed
to acquire network knowledge, aiding in topology decisions. Third, a multipath
auxiliary transmission mechanism is introduced to enhance network awareness and
facilitate multipath transmissions. Lastly, we design policy consistency
protocols to guarantee seamless updates of transmission policies. Empirical
results demonstrate that NetStorm significantly outperforms distributed
training systems like MXNET, MLNET, and TSEngine, with a speedup of 6.5~9.2
times over MXNET.
