  Model inversion (MI) attacks aim to reveal sensitive information in training
datasets by solely accessing model weights. Generative MI attacks, a prominent
strand in this field, utilize auxiliary datasets to recreate target data
attributes, restricting the images to remain photo-realistic, but their success
often depends on the similarity between auxiliary and target datasets. If the
distributions are dissimilar, existing MI attack attempts frequently fail,
yielding unrealistic or target-unrelated results. In response to these
challenges, we introduce a groundbreaking approach named Patch-MI, inspired by
jigsaw puzzle assembly. To this end, we build upon a new probabilistic
interpretation of MI attacks, employing a generative adversarial network
(GAN)-like framework with a patch-based discriminator. This approach allows the
synthesis of images that are similar to the target dataset distribution, even
in cases of dissimilar auxiliary dataset distribution. Moreover, we artfully
employ a random transformation block, a sophisticated maneuver that crafts
generalized images, thus enhancing the efficacy of the target classifier. Our
numerical and graphical findings demonstrate that Patch-MI surpasses existing
generative MI methods in terms of accuracy, marking significant advancements
while preserving comparable statistical dataset quality. For reproducibility of
our results, we make our source code publicly available in
https://github.com/jonggyujang0123/Patch-Attack.
