  A persistent challenge in conditional image synthesis has been to generate
diverse output images from the same input image despite only one output image
being observed per input image. GAN-based methods are prone to mode collapse,
which leads to low diversity. To get around this, we leverage Implicit Maximum
Likelihood Estimation (IMLE) which can overcome mode collapse fundamentally.
IMLE uses the same generator as GANs but trains it with a different,
non-adversarial objective which ensures each observed image has a generated
sample nearby. Unfortunately, to generate high-fidelity images, prior
IMLE-based methods require a large number of samples, which is expensive. In
this paper, we propose a new method to get around this limitation, which we dub
Conditional Hierarchical IMLE (CHIMLE), which can generate high-fidelity images
without requiring many samples. We show CHIMLE significantly outperforms the
prior best IMLE, GAN and diffusion-based methods in terms of image fidelity and
mode coverage across four tasks, namely night-to-day, 16x single image
super-resolution, image colourization and image decompression. Quantitatively,
our method improves Fr\'echet Inception Distance (FID) by 36.9% on average
compared to the prior best IMLE-based method, and by 27.5% on average compared
to the best non-IMLE-based general-purpose methods.
