  Conditional visual generation has witnessed remarkable progress with the
advent of diffusion models (DMs), especially in tasks like control-to-image
generation. However, challenges such as expensive computational cost, high
inference latency, and difficulties of integration with large language models
(LLMs) have necessitated exploring alternatives to DMs. This paper introduces
ControlVAR, a novel framework that explores pixel-level controls in visual
autoregressive (VAR) modeling for flexible and efficient conditional
generation. In contrast to traditional conditional models that learn the
conditional distribution, ControlVAR jointly models the distribution of image
and pixel-level conditions during training and imposes conditional controls
during testing. To enhance the joint modeling, we adopt the next-scale AR
prediction paradigm and unify control and image representations. A
teacher-forcing guidance strategy is proposed to further facilitate
controllable generation with joint modeling. Extensive experiments demonstrate
the superior efficacy and flexibility of ControlVAR across various conditional
generation tasks against popular conditional DMs, \eg, ControlNet and
T2I-Adaptor.
