  Empirical risk minimization frequently employs convex surrogates to
underlying discrete loss functions in order to achieve computational
tractability during optimization. However, classical convex surrogates can only
tightly bound modular loss functions, sub-modular functions or supermodular
functions separately while maintaining polynomial time computation. In this
work, a novel generic convex surrogate for general non-modular loss functions
is introduced, which provides for the first time a tractable solution for loss
functions that are neither super-modular nor submodular. This convex surro-gate
is based on a submodular-supermodular decomposition for which the existence and
uniqueness is proven in this paper. It takes the sum of two convex surrogates
that separately bound the supermodular component and the submodular component
using slack-rescaling and the Lov{\'a}sz hinge, respectively. It is further
proven that this surrogate is convex , piecewise linear, an extension of the
loss function, and for which subgradient computation is polynomial time.
Empirical results are reported on a non-submodular loss based on the
S{{\o}}rensen-Dice difference function, and a real-world face track dataset
with tens of thousands of frames, demonstrating the improved performance,
efficiency, and scalabil-ity of the novel convex surrogate.
