  We consider the distributed SGD problem, where a main node distributes
gradient calculations among $n$ workers. By assigning tasks to all the workers
and waiting only for the $k$ fastest ones, the main node can trade-off the
algorithm's error with its runtime by gradually increasing $k$ as the algorithm
evolves. However, this strategy, referred to as adaptive $k$-sync, neglects the
cost of unused computations and of communicating models to workers that reveal
a straggling behavior. We propose a cost-efficient scheme that assigns tasks
only to $k$ workers, and gradually increases $k$. We introduce the use of a
combinatorial multi-armed bandit model to learn which workers are the fastest
while assigning gradient calculations. Assuming workers with exponentially
distributed response times parameterized by different means, we give empirical
and theoretical guarantees on the regret of our strategy, i.e., the extra time
spent to learn the mean response times of the workers. Furthermore, we propose
and analyze a strategy applicable to a large class of response time
distributions. Compared to adaptive $k$-sync, our scheme achieves significantly
lower errors with the same computational efforts and less downlink
communication while being inferior in terms of speed.
