  Recent developments in large-scale pre-trained text-to-image diffusion models
have significantly improved the generation of high-fidelity images,
particularly with the emergence of diffusion models based on transformer
architecture (DiTs). Among these diffusion models, diffusion transformers have
demonstrated superior image generation capabilities, boosting lower FID scores
and higher scalability. However, deploying large-scale DiT models can be
expensive due to their extensive parameter numbers. Although existing research
has explored efficient deployment techniques for diffusion models such as model
quantization, there is still little work concerning DiT-based models. To tackle
this research gap, in this paper, we propose TerDiT, a quantization-aware
training (QAT) and efficient deployment scheme for ternary diffusion models
with transformers. We focus on the ternarization of DiT networks and scale
model sizes from 600M to 4.2B. Our work contributes to the exploration of
efficient deployment strategies for large-scale DiT models, demonstrating the
feasibility of training extremely low-bit diffusion transformer models from
scratch while maintaining competitive image generation capacities compared to
full-precision models. Code will be available at
https://github.com/Lucky-Lance/TerDiT.
