  This paper investigates the distinctions between gradient methods applied to
non-differentiable functions (NGDMs) and classical gradient descents (GDs)
designed for differentiable functions. First, we demonstrate significant
differences in the convergence properties of NGDMs compared to GDs, challenging
the applicability of the extensive neural network convergence literature based
on $L-smoothness$ to non-smooth neural networks. Next, we demonstrate the
paradoxical nature of NGDM solutions for $L_{1}$-regularized problems, showing
that increasing the regularization penalty leads to an increase in the $L_{1}$
norm of optimal solutions in NGDMs. Consequently, we show that widely adopted
$L_{1}$ penalization-based techniques for network pruning do not yield expected
results. Additionally, we dispel the common belief that optimization algorithms
like Adam and RMSProp perform similarly in non-differentiable contexts.
Finally, we explore the Edge of Stability phenomenon, indicating its
inapplicability even to Lipschitz continuous convex differentiable functions,
leaving its relevance to non-convex non-differentiable neural networks
inconclusive. Our analysis exposes misguided interpretations of NGDMs in widely
referenced papers and texts due to an overreliance on strong smoothness
assumptions, emphasizing the necessity for a nuanced understanding of
foundational assumptions in the analysis of these systems.
