  Deep neural networks trained end-to-end to map a measurement of a (noisy)
image to a clean image perform excellent for a variety of linear inverse
problems. Current methods are only trained on a few hundreds or thousands of
images as opposed to the millions of examples deep networks are trained on in
other domains. In this work, we study whether major performance gains are
expected from scaling up the training set size. We consider image denoising,
accelerated magnetic resonance imaging, and super-resolution and empirically
determine the reconstruction quality as a function of training set size, while
simultaneously scaling the network size. For all three tasks we find that an
initially steep power-law scaling slows significantly already at moderate
training set sizes. Interpolating those scaling laws suggests that even
training on millions of images would not significantly improve performance. To
understand the expected behavior, we analytically characterize the performance
of a linear estimator learned with early stopped gradient descent. The result
formalizes the intuition that once the error induced by learning the signal
model is small relative to the error floor, more training examples do not
improve performance.
