  Emerging applications of machine learning in numerous areas involve
continuous gathering of and learning from streams of data. Real-time
incorporation of streaming data into the learned models is essential for
improved inference in these applications. Further, these applications often
involve data that are either inherently gathered at geographically distributed
entities or that are intentionally distributed across multiple machines for
memory, computational, and/or privacy reasons. Training of models in this
distributed, streaming setting requires solving stochastic optimization
problems in a collaborative manner over communication links between the
physical entities. When the streaming data rate is high compared to the
processing capabilities of compute nodes and/or the rate of the communications
links, this poses a challenging question: how can one best leverage the
incoming data for distributed training under constraints on computing
capabilities and/or communications rate? A large body of research has emerged
in recent decades to tackle this and related problems. This paper reviews
recently developed methods that focus on large-scale distributed stochastic
optimization in the compute- and bandwidth-limited regime, with an emphasis on
convergence analysis that explicitly accounts for the mismatch between
computation, communication and streaming rates. In particular, it focuses on
methods that solve: (i) distributed stochastic convex problems, and (ii)
distributed principal component analysis, which is a nonconvex problem with
geometric structure that permits global convergence. For such methods, the
paper discusses recent advances in terms of distributed algorithmic designs
when faced with high-rate streaming data. Further, it reviews guarantees
underlying these methods, which show there exist regimes in which systems can
learn from distributed, streaming data at order-optimal rates.
