  In practical distributed systems, workers are typically not homogeneous, and
due to differences in hardware configurations and network conditions, can have
highly varying processing times. We consider smooth nonconvex finite-sum
(empirical risk minimization) problems in this setup and introduce a new
parallel method, Freya PAGE, designed to handle arbitrarily heterogeneous and
asynchronous computations. By being robust to "stragglers" and adaptively
ignoring slow computations, Freya PAGE offers significantly improved time
complexity guarantees compared to all previous methods, including Asynchronous
SGD, Rennala SGD, SPIDER, and PAGE, while requiring weaker assumptions. The
algorithm relies on novel generic stochastic gradient collection strategies
with theoretical guarantees that can be of interest on their own, and may be
used in the design of future optimization methods. Furthermore, we establish a
lower bound for smooth nonconvex finite-sum problems in the asynchronous setup,
providing a fundamental time complexity limit. This lower bound is tight and
demonstrates the optimality of Freya PAGE in the large-scale regime, i.e., when
$\sqrt{m} \geq n$, where $n$ is # of workers, and $m$ is # of data samples.
