  Despite the impressive performance of Deep Neural Networks (DNNs) on various
vision tasks, they still exhibit erroneous high sensitivity toward semantic
primitives (e.g. object pose). We propose a theoretically grounded analysis for
DNN robustness in the semantic space. We qualitatively analyze different DNNs'
semantic robustness by visualizing the DNN global behavior as semantic maps and
observe interesting behavior of some DNNs. Since generating these semantic maps
does not scale well with the dimensionality of the semantic space, we develop a
bottom-up approach to detect robust regions of DNNs. To achieve this, we
formalize the problem of finding robust semantic regions of the network as
optimizing integral bounds and we develop expressions for update directions of
the region bounds. We use our developed formulations to quantitatively evaluate
the semantic robustness of different popular network architectures. We show
through extensive experimentation that several networks, while trained on the
same dataset and enjoying comparable accuracy, do not necessarily perform
similarly in semantic robustness. For example, InceptionV3 is more accurate
despite being less semantically robust than ResNet50. We hope that this tool
will serve as a milestone towards understanding the semantic robustness of
DNNs.
