  Neural network (NN) denoisers are an essential building block in many common
tasks, ranging from image reconstruction to image generation. However, the
success of these models is not well understood from a theoretical perspective.
In this paper, we aim to characterize the functions realized by shallow ReLU NN
denoisers -- in the common theoretical setting of interpolation (i.e., zero
training loss) with a minimal representation cost (i.e., minimal $\ell^2$ norm
weights). First, for univariate data, we derive a closed form for the NN
denoiser function, find it is contractive toward the clean data points, and
prove it generalizes better than the empirical MMSE estimator at a low noise
level. Next, for multivariate data, we find the NN denoiser functions in a
closed form under various geometric assumptions on the training data: data
contained in a low-dimensional subspace, data contained in a union of one-sided
rays, or several types of simplexes. These functions decompose into a sum of
simple rank-one piecewise linear interpolations aligned with edges and/or faces
connecting training samples. We empirically verify this alignment phenomenon on
synthetic data and real images.
