  In this paper, we provide an in-depth study of Stochastic Backpropagation
(SBP) when training deep neural networks for standard image classification and
object detection tasks. During backward propagation, SBP calculates the
gradients by only using a subset of feature maps to save the GPU memory and
computational cost. We interpret SBP as an efficient way to implement
stochastic gradient decent by performing backpropagation dropout, which leads
to considerable memory saving and training process speedup, with a minimal
impact on the overall model accuracy. We offer some good practices to apply SBP
in training image recognition models, which can be adopted in learning a wide
range of deep neural networks. Experiments on image classification and object
detection show that SBP can save up to 40% of GPU memory with less than 1%
accuracy degradation.
