  StableDiffusion is a revolutionary text-to-image generator that is causing a
stir in the world of image generation and editing. Unlike traditional methods
that learn a diffusion model in pixel space, StableDiffusion learns a diffusion
model in the latent space via a VQGAN, ensuring both efficiency and quality. It
not only supports image generation tasks, but also enables image editing for
real images, such as image inpainting and local editing. However, we have
observed that the vanilla VQGAN used in StableDiffusion leads to significant
information loss, causing distortion artifacts even in non-edited image
regions. To this end, we propose a new asymmetric VQGAN with two simple
designs. Firstly, in addition to the input from the encoder, the decoder
contains a conditional branch that incorporates information from task-specific
priors, such as the unmasked image region in inpainting. Secondly, the decoder
is much heavier than the encoder, allowing for more detailed recovery while
only slightly increasing the total inference cost. The training cost of our
asymmetric VQGAN is cheap, and we only need to retrain a new asymmetric decoder
while keeping the vanilla VQGAN encoder and StableDiffusion unchanged. Our
asymmetric VQGAN can be widely used in StableDiffusion-based inpainting and
local editing methods. Extensive experiments demonstrate that it can
significantly improve the inpainting and editing performance, while maintaining
the original text-to-image capability. The code is available at
\url{https://github.com/buxiangzhiren/Asymmetric_VQGAN}.
