  We introduce a robust algorithm for face verification, i.e., deciding whether
twoimages are of the same person or not. Our approach is a novel take on the
idea ofusing deep generative networks for adversarial robustness. We use the
generativemodel during training as an online augmentation method instead of a
test-timepurifier that removes adversarial noise. Our architecture uses a
contrastive loss termand a disentangled generative model to sample negative
pairs. Instead of randomlypairing two real images, we pair an image with its
class-modified counterpart whilekeeping its content (pose, head tilt, hair,
etc.) intact. This enables us to efficientlysample hard negative pairs for the
contrastive loss. We experimentally show that, when coupled with adversarial
training, the proposed scheme converges with aweak inner solver and has a
higher clean and robust accuracy than state-of-the-art-methods when evaluated
against white-box physical attacks.
