  We formalize and attack the problem of generating new images from old ones
that are as diverse as possible, only allowing them to change without
restrictions in certain parts of the image while remaining globally consistent.
This encompasses the typical situation found in generative modelling, where we
are happy with parts of the generated data, but would like to resample others
("I like this generated castle overall, but this tower looks unrealistic, I
would like a new one"). In order to attack this problem we build from the best
conditional and unconditional generative models to introduce a new network
architecture, training procedure, and algorithm for resampling parts of the
image as desired.
