  It is generally thought that the use of stochastic activation functions in
deep learning architectures yield models with superior generalization
abilities. However, a sufficiently rigorous statement and theoretical proof of
this heuristic is lacking in the literature. In this paper, we provide several
novel contributions to the literature in this regard. Defining a new notion of
nonlocal directional derivative, we analyze its theoretical properties
(existence and convergence). Second, using a probabilistic reformulation, we
show that nonlocal derivatives are epsilon-sub gradients, and derive sample
complexity results for convergence of stochastic gradient descent-like methods
using nonlocal derivatives. Finally, using our analysis of the nonlocal
gradient of Holder continuous functions, we observe that sample paths of
Brownian motion admit nonlocal directional derivatives, and the nonlocal
derivatives of Brownian motion are seen to be Gaussian processes with
computable mean and standard deviation. Using the theory of nonlocal
directional derivatives, we solve a highly nondifferentiable and nonconvex
model problem of parameter estimation on image articulation manifolds. Using
Brownian motion infused ReLU activation functions with the nonlocal gradient in
place of the usual gradient during backpropagation, we also perform experiments
on multiple well-studied deep learning architectures. Our experiments indicate
the superior generalization capabilities of Brownian neural activation
functions in low-training data regimes, where the use of stochastic neurons
beats the deterministic ReLU counterpart.
