  Large foundation models, such as large language models, have performed
exceptionally well in various application scenarios. Building or fully
fine-tuning such large models is usually prohibitive due to either hardware
budget or lack of access to backpropagation. The zeroth-order methods offer a
promising direction for tackling this challenge, where only forward passes are
needed to update the model. This paper introduces an efficient Stochastic
Two-Point (S2P) approach within the gradient-free regime. We present the
theoretical convergence properties of S2P under the general and relaxed
smoothness assumptions, and the derived results help understand and inherently
connect the two popular types of zeroth-order methods, basic random search and
stochastic three-point method. The theoretical properties also shed light on a
Variant of S2P (VS2P), through exploiting our new convergence properties that
better represent the dynamics of deep models in training. Our comprehensive
empirical results show that VS2P is highly effective in optimizing objectives
for deep models. It outperforms or achieves competitive performance compared to
standard methods across various model types and scales.
