  We consider a variant of the stochastic gradient descent (SGD) with a random
learning rate and reveal its convergence properties. SGD is a widely used
stochastic optimization algorithm in machine learning, especially deep
learning. Numerous studies reveal the convergence properties of SGD and its
simplified variants. Among these, the analysis of convergence using a
stationary distribution of updated parameters provides generalizable results.
However, to obtain a stationary distribution, the update direction of the
parameters must not degenerate, which limits the applicable variants of SGD. In
this study, we consider a novel SGD variant, Poisson SGD, which has degenerated
parameter update directions and instead utilizes a random learning rate.
Consequently, we demonstrate that a distribution of a parameter updated by
Poisson SGD converges to a stationary distribution under weak assumptions on a
loss function. Based on this, we further show that Poisson SGD finds global
minima in non-convex optimization problems and also evaluate the generalization
error using this method. As a proof technique, we approximate the distribution
by Poisson SGD with that of the bouncy particle sampler (BPS) and derive its
stationary distribution, using the theoretical advance of the piece-wise
deterministic Markov process (PDMP).
