  We propose an alternative approach to neural network training using the
monotone vector field, an idea inspired by the seminal work of Juditsky and
Nemirovski [Juditsky & Nemirovsky, 2019] developed originally to solve
parameter estimation problems for generalized linear models (GLM) by reducing
the original non-convex problem to a convex problem of solving a monotone
variational inequality (VI). Our approach leads to computationally efficient
procedures that converge fast and offer guarantee in some special cases, such
as training a single-layer neural network or fine-tuning the last layer of the
pre-trained model. Our approach can be used for more efficient fine-tuning of a
pre-trained model while freezing the bottom layers, an essential step for
deploying many machine learning models such as large language models (LLM). We
demonstrate its applicability in training fully-connected (FC) neural networks,
graph neural networks (GNN), and convolutional neural networks (CNN) and show
the competitive or better performance of our approach compared to stochastic
gradient descent methods on both synthetic and real network data prediction
tasks regarding various performance metrics.
