  The impressive performance of deep neural networks (DNNs) has immensely
strengthened the line of research that aims at theoretically analyzing their
effectiveness. This has incited research on the reaction of DNNs to noisy
input, namely developing adversarial input attacks and strategies that lead to
robust DNNs to these attacks. To that end, in this paper, we derive exact
analytic expressions for the first and second moments (mean and variance) of a
small piecewise linear (PL) network (Affine, ReLU, Affine) subject to Gaussian
input. In particular, we generalize the second-moment expression of Bibi et al.
to arbitrary input Gaussian distributions, dropping the zero-mean assumption.
We show that the new variance expression can be efficiently approximated
leading to much tighter variance estimates as compared to the preliminary
results of Bibi et al. Moreover, we experimentally show that these expressions
are tight under simple linearizations of deeper PL-DNNs, where we investigate
the effect of the linearization sensitivity on the accuracy of the moment
estimates. Lastly, we show that the derived expressions can be used to
construct sparse and smooth Gaussian adversarial attacks (targeted and
non-targeted) that tend to lead to perceptually feasible input attacks.
