  The most advanced diffusion models have recently adopted increasingly deep
stacked networks (e.g., U-Net or Transformer) to promote the generative
emergence capabilities of vision generation models similar to large language
models (LLMs). However, progressively deeper stacked networks will intuitively
cause numerical propagation errors and reduce noisy prediction capabilities on
generative data, which hinders massively deep scalable training of vision
generation models. In this paper, we first uncover the nature that neural
networks being able to effectively perform generative denoising lies in the
fact that the intrinsic residual unit has consistent dynamic property with the
input signal's reverse diffusion process, thus supporting excellent generative
abilities. Afterwards, we stand on the shoulders of two common types of deep
stacked networks to propose a unified and massively scalable Neural Residual
Diffusion Models framework (Neural-RDM for short), which is a simple yet
meaningful change to the common architecture of deep generative networks by
introducing a series of learnable gated residual parameters that conform to the
generative dynamics. Experimental results on various generative tasks show that
the proposed neural residual models obtain state-of-the-art scores on image's
and video's generative benchmarks. Rigorous theoretical proofs and extensive
experiments also demonstrate the advantages of this simple gated residual
mechanism consistent with dynamic modeling in improving the fidelity and
consistency of generated content and supporting large-scale scalable training.
Code is available at https://github.com/Anonymous/Neural-RDM.
