  Diffusion models with large-scale pre-training have achieved significant
success in the field of visual content generation, particularly exemplified by
Diffusion Transformers (DiT). However, DiT models have faced challenges with
scalability and quadratic complexity efficiency. In this paper, we aim to
leverage the long sequence modeling capability of Gated Linear Attention (GLA)
Transformers, expanding its applicability to diffusion models. We introduce
Diffusion Gated Linear Attention Transformers (DiG), a simple, adoptable
solution with minimal parameter overhead, following the DiT design, but
offering superior efficiency and effectiveness. In addition to better
performance than DiT, DiG-S/2 exhibits $2.5\times$ higher training speed than
DiT-S/2 and saves $75.7\%$ GPU memory at a resolution of $1792 \times 1792$.
Moreover, we analyze the scalability of DiG across a variety of computational
complexity. DiG models, with increased depth/width or augmentation of input
tokens, consistently exhibit decreasing FID. We further compare DiG with other
subquadratic-time diffusion models. With the same model size, DiG-XL/2 is
$4.2\times$ faster than the recent Mamba-based diffusion model at a $1024$
resolution, and is $1.8\times$ faster than DiT with CUDA-optimized
FlashAttention-2 under the $2048$ resolution. All these results demonstrate its
superior efficiency among the latest diffusion models. Code is released at
https://github.com/hustvl/DiG.
