  Stochastic versions of proximal methods have gained much attention in
statistics and machine learning. These algorithms tend to admit simple,
scalable forms, and enjoy numerical stability via implicit updates. In this
work, we propose and analyze a stochastic version of the recently proposed
proximal distance algorithm, a class of iterative optimization methods that
recover a desired constrained estimation problem as a penalty parameter $\rho
\rightarrow \infty$. By uncovering connections to related stochastic proximal
methods and interpreting the penalty parameter as the learning rate, we justify
heuristics used in practical manifestations of the proximal distance method,
establishing their convergence guarantees for the first time. Moreover, we
extend recent theoretical devices to establish finite error bounds and a
complete characterization of convergence rates regimes. We validate our
analysis via a thorough empirical study, also showing that unsurprisingly, the
proposed method outpaces batch versions on popular learning tasks.
