  Existing vector quantization (VQ) based autoregressive models follow a
two-stage generation paradigm that first learns a codebook to encode images as
discrete codes, and then completes generation based on the learned codebook.
However, they encode fixed-size image regions into fixed-length codes and
ignore their naturally different information densities, which results in
insufficiency in important regions and redundancy in unimportant ones, and
finally degrades the generation quality and speed. Moreover, the fixed-length
coding leads to an unnatural raster-scan autoregressive generation. To address
the problem, we propose a novel two-stage framework: (1) Dynamic-Quantization
VAE (DQ-VAE) which encodes image regions into variable-length codes based on
their information densities for an accurate and compact code representation.
(2) DQ-Transformer which thereby generates images autoregressively from
coarse-grained (smooth regions with fewer codes) to fine-grained (details
regions with more codes) by modeling the position and content of codes in each
granularity alternately, through a novel stacked-transformer architecture and
shared-content, non-shared position input layers designs. Comprehensive
experiments on various generation tasks validate our superiorities in both
effectiveness and efficiency. Code will be released at
https://github.com/CrossmodalGroup/DynamicVectorQuantization.
