  Adversarial patch attacks present a significant threat to real-world object
detectors due to their practical feasibility. Existing defense methods, which
rely on attack data or prior knowledge, struggle to effectively address a wide
range of adversarial patches. In this paper, we show two inherent
characteristics of adversarial patches, semantic independence and spatial
heterogeneity, independent of their appearance, shape, size, quantity, and
location. Semantic independence indicates that adversarial patches operate
autonomously within their semantic context, while spatial heterogeneity
manifests as distinct image quality of the patch area that differs from
original clean image due to the independent generation process. Based on these
observations, we propose PAD, a novel adversarial patch localization and
removal method that does not require prior knowledge or additional training.
PAD offers patch-agnostic defense against various adversarial patches,
compatible with any pre-trained object detectors. Our comprehensive digital and
physical experiments involving diverse patch types, such as localized noise,
printable, and naturalistic patches, exhibit notable improvements over
state-of-the-art works. Our code is available at
https://github.com/Lihua-Jing/PAD.
