  Soft random sampling (SRS) is a simple yet effective approach for efficient
training of large-scale deep neural networks when dealing with massive data.
SRS selects a subset uniformly at random with replacement from the full data
set in each epoch. In this paper, we conduct a theoretical and empirical
analysis of SRS. First, we analyze its sampling dynamics including data
coverage and occupancy. Next, we investigate its convergence with non-convex
objective functions and give the convergence rate. Finally, we provide its
generalization performance. We empirically evaluate SRS for image recognition
on CIFAR10 and automatic speech recognition on Librispeech and an in-house
payload dataset to demonstrate its effectiveness. Compared to existing
coreset-based data selection methods, SRS offers a better accuracy-efficiency
trade-off. Especially on real-world industrial scale data sets, it is shown to
be a powerful training strategy with significant speedup and competitive
performance with almost no additional computing cost.
