  Lossy image compression aims to represent images in as few bits as possible
while maintaining fidelity to the original. Theoretical results indicate that
optimizing distortion metrics such as PSNR or MS-SSIM necessarily leads to a
discrepancy in the statistics of original images from those of reconstructions,
in particular at low bitrates, often manifested by the blurring of the
compressed images. Previous work has leveraged adversarial discriminators to
improve statistical fidelity. Yet these binary discriminators adopted from
generative modeling tasks may not be ideal for image compression. In this
paper, we introduce a non-binary discriminator that is conditioned on quantized
local image representations obtained via VQ-VAE autoencoders. Our evaluations
on the CLIC2020, DIV2K and Kodak datasets show that our discriminator is more
effective for jointly optimizing distortion (e.g., PSNR) and statistical
fidelity (e.g., FID) than the PatchGAN of the state-of-the-art HiFiC model. On
CLIC2020, we obtain the same FID as HiFiC with 30-40\% fewer bits.
