  A deep generative model such as a GAN learns to model a rich set of semantic
and physical rules about the target distribution, but up to now, it has been
obscure how such rules are encoded in the network, or how a rule could be
changed. In this paper, we introduce a new problem setting: manipulation of
specific rules encoded by a deep generative model. To address the problem, we
propose a formulation in which the desired rule is changed by manipulating a
layer of a deep network as a linear associative memory. We derive an algorithm
for modifying one entry of the associative memory, and we demonstrate that
several interesting structural rules can be located and modified within the
layers of state-of-the-art generative models. We present a user interface to
enable users to interactively change the rules of a generative model to achieve
desired effects, and we show several proof-of-concept applications. Finally,
results on multiple datasets demonstrate the advantage of our method against
standard fine-tuning methods and edit transfer algorithms.
