  We study deterministic and stochastic primal-dual sub-gradient algorithms for
distributed optimization of a separable objective function with global
inequality constraints. In both algorithms, the norm of the Lagrangian
multipliers are controlled by augmenting the corresponding Lagrangian function
with a quadratic regularization term. Specifically, we show that when the
stepsize of each algorithm satisfies a certain restriction, the norm of the
Lagrangian multipliers is upper bounded by an expression that is inversely
proportional to the parameter of the regularization. We use this result to
compute upper bounds on the sub-gradients of the Lagrangian function. For the
deterministic algorithm, we prove a convergence rate for attaining the optimal
objective value. In the stochastic optimization case, we similarly prove
convergence rates both in the expectation and with a high probability, using
the method of bounded martingale difference. For both algorithms, we
demonstrate a trade-off between the convergence rate and the decay rate of the
constraint violation, in the sense that improving the convergence rate slows
the decay rate of the constraint violation and vice versa. We demonstrate the
convergence of our proposed algorithms numerically for distributed regression
with the hinge and logistic loss functions over different graph structures.
