  Recent developments in diffusion models have advanced conditioned image
generation, yet they struggle with reconstructing out-of-distribution (OOD)
images, such as unseen tumors in medical images, causing "image hallucination"
and risking misdiagnosis. We hypothesize such hallucinations result from local
OOD regions in the conditional images. We verify that partitioning the OOD
region and conducting separate image generations alleviates hallucinations in
several applications. From this, we propose a training-free diffusion framework
that reduces hallucination with multiple Local Diffusion processes. Our
approach involves OOD estimation followed by two modules: a "branching" module
generates locally both within and outside OOD regions, and a "fusion" module
integrates these predictions into one. Our evaluation shows our method
mitigates hallucination over baseline models quantitatively and qualitatively,
reducing misdiagnosis by 40% and 25% in the real-world medical and natural
image datasets, respectively. It also demonstrates compatibility with various
pre-trained diffusion models.
