  In this work, we consider nonconvex composite problems that involve
inf-convolution with a Legendre function, which gives rise to an anisotropic
generalization of the proximal mapping and Moreau-envelope. In a convex setting
such problems can be solved via alternating minimization of a splitting
formulation, where the consensus constraint is penalized with a Legendre
function. In contrast, for nonconvex models it is in general unclear that this
approach yields stationary points to the infimal convolution problem. To this
end we analytically investigate local regularity properties of the
Moreau-envelope function under prox-regularity, which allows us to establish
the equivalence between stationary points of the splitting model and the
original inf-convolution model. We apply our theory to characterize stationary
points of the penalty objective, which is minimized by the elastic averaging
SGD (EASGD) method for distributed training. Numerically, we demonstrate the
practical relevance of the proposed approach on the important task of
distributed training of deep neural networks.
