  The adversarial risk of a machine learning model has been widely studied.
Most previous works assume that the data lies in the whole ambient space. We
propose to take a new angle and take the manifold assumption into
consideration. Assuming data lies in a manifold, we investigate two new types
of adversarial risk, the normal adversarial risk due to perturbation along
normal direction, and the in-manifold adversarial risk due to perturbation
within the manifold. We prove that the classic adversarial risk can be bounded
from both sides using the normal and in-manifold adversarial risks. We also
show with a surprisingly pessimistic case that the standard adversarial risk
can be nonzero even when both normal and in-manifold risks are zero. We
finalize the paper with empirical studies supporting our theoretical results.
Our results suggest the possibility of improving the robustness of a classifier
by only focusing on the normal adversarial risk.
