  In this paper, we propose a novel optimization algorithm for training machine
learning models called Input Normalized Stochastic Gradient Descent (INSGD),
inspired by the Normalized Least Mean Squares (NLMS) algorithm used in adaptive
filtering. When training complex models on large datasets, the choice of
optimizer parameters, particularly the learning rate, is crucial to avoid
divergence. Our algorithm updates the network weights using stochastic gradient
descent with $\ell_1$ and $\ell_2$-based normalizations applied to the learning
rate, similar to NLMS. However, unlike existing normalization methods, we
exclude the error term from the normalization process and instead normalize the
update term using the input vector to the neuron. Our experiments demonstrate
that our optimization algorithm achieves higher accuracy levels compared to
different initialization settings. We evaluate the efficiency of our training
algorithm on benchmark datasets using ResNet-18, WResNet-20, ResNet-50, and a
toy neural network. Our INSGD algorithm improves the accuracy of ResNet-18 on
CIFAR-10 from 92.42\% to 92.71\%, WResNet-20 on CIFAR-100 from 76.20\% to
77.39\%, and ResNet-50 on ImageNet-1K from 75.52\% to 75.67\%.
