  Denoising diffusion probabilistic models for image inpainting aim to add the
noise to the texture of image during the forward process and recover masked
regions with unmasked ones of the texture via the reverse denoising process.
Despite the meaningful semantics generation, the existing arts suffer from the
semantic discrepancy between masked and unmasked regions, since the
semantically dense unmasked texture fails to be completely degraded while the
masked regions turn to the pure noise in diffusion process, leading to the
large discrepancy between them. In this paper, we aim to answer how unmasked
semantics guide texture denoising process;together with how to tackle the
semantic discrepancy, to facilitate the consistent and meaningful semantics
generation. To this end, we propose a novel structure-guided diffusion model
named StrDiffusion, to reformulate the conventional texture denoising process
under structure guidance to derive a simplified denoising objective for image
inpainting, while revealing: 1) the semantically sparse structure is beneficial
to tackle semantic discrepancy in early stage, while dense texture generates
reasonable semantics in late stage; 2) the semantics from unmasked regions
essentially offer the time-dependent structure guidance for the texture
denoising process, benefiting from the time-dependent sparsity of the structure
semantics. For the denoising process, a structure-guided neural network is
trained to estimate the simplified denoising objective by exploiting the
consistency of the denoised structure between masked and unmasked regions.
Besides, we devise an adaptive resampling strategy as a formal criterion as
whether structure is competent to guide the texture denoising process, while
regulate their semantic correlations. Extensive experiments validate the merits
of StrDiffusion over the state-of-the-arts. Our code is available at
https://github.com/htyjers/StrDiffusion.
