  We present a randomized forward mode gradient (RFG) as an alternative to
backpropagation. RFG is a random estimator for the gradient that is constructed
based on the directional derivative along a random vector. The forward mode
automatic differentiation (AD) provides an efficient computation of RFG. The
probability distribution of the random vector determines the statistical
properties of RFG. Through the second moment analysis, we found that the
distribution with the smallest kurtosis yields the smallest expected relative
squared error. By replacing gradient with RFG, a class of RFG-based
optimization algorithms is obtained. By focusing on gradient descent (GD) and
Polyak's heavy ball (PHB) methods, we present a convergence analysis of
RFG-based optimization algorithms for quadratic functions. Computational
experiments are presented to demonstrate the performance of the proposed
algorithms and verify the theoretical findings.
