  Gradient-only line searches (GOLS) adaptively determine step sizes along
search directions for discontinuous loss functions resulting from dynamic
mini-batch sub-sampling in neural network training. Step sizes in GOLS are
determined by localizing Stochastic Non-Negative Associated Gradient Projection
Points (SNN-GPPs) along descent directions. These are identified by a sign
change in the directional derivative from negative to positive along a descent
direction. Activation functions are a significant component of neural network
architectures as they introduce non-linearities essential for complex function
approximations. The smoothness and continuity characteristics of the activation
functions directly affect the gradient characteristics of the loss function to
be optimized. Therefore, it is of interest to investigate the relationship
between activation functions and different neural network architectures in the
context of GOLS. We find that GOLS are robust for a range of activation
functions, but sensitive to the Rectified Linear Unit (ReLU) activation
function in standard feedforward architectures. The zero-derivative in ReLU's
negative input domain can lead to the gradient-vector becoming sparse, which
severely affects training. We show that implementing architectural features
such as batch normalization and skip connections can alleviate these
difficulties and benefit training with GOLS for all activation functions
considered.
