  Injectivity plays an important role in generative models where it enables
inference; in inverse problems and compressed sensing with generative priors it
is a precursor to well posedness. We establish sharp characterizations of
injectivity of fully-connected and convolutional ReLU layers and networks.
First, through a layerwise analysis, we show that an expansivity factor of two
is necessary and sufficient for injectivity by constructing appropriate weight
matrices. We show that global injectivity with iid Gaussian matrices, a
commonly used tractable model, requires larger expansivity between 3.4 and
10.5. We also characterize the stability of inverting an injective network via
worst-case Lipschitz constants of the inverse. We then use arguments from
differential topology to study injectivity of deep networks and prove that any
Lipschitz map can be approximated by an injective ReLU network. Finally, using
an argument based on random projections, we show that an end-to-end -- rather
than layerwise -- doubling of the dimension suffices for injectivity. Our
results establish a theoretical basis for the study of nonlinear inverse and
inference problems using neural networks.
