  Deep learning models suffer from the problem of semantic discontinuity: small
perturbations in the input space tend to cause semantic-level interference to
the model output. We argue that the semantic discontinuity results from these
inappropriate training targets and contributes to notorious issues such as
adversarial robustness, interpretability, etc. We first conduct data analysis
to provide evidence of semantic discontinuity in existing deep learning models,
and then design a simple semantic continuity constraint which theoretically
enables models to obtain smooth gradients and learn semantic-oriented features.
Qualitative and quantitative experiments prove that semantically continuous
models successfully reduce the use of non-semantic information, which further
contributes to the improvement in adversarial robustness, interpretability,
model transfer, and machine bias.
