  Portrait harmonization aims to composite a subject into a new background,
adjusting its lighting and color to ensure harmony with the background scene.
Existing harmonization techniques often only focus on adjusting the global
color and brightness of the foreground and ignore crucial illumination cues
from the background such as apparent lighting direction, leading to unrealistic
compositions. We introduce Relightful Harmonization, a lighting-aware diffusion
model designed to seamlessly harmonize sophisticated lighting effect for the
foreground portrait using any background image. Our approach unfolds in three
stages. First, we introduce a lighting representation module that allows our
diffusion model to encode lighting information from target image background.
Second, we introduce an alignment network that aligns lighting features learned
from image background with lighting features learned from panorama environment
maps, which is a complete representation for scene illumination. Last, to
further boost the photorealism of the proposed method, we introduce a novel
data simulation pipeline that generates synthetic training pairs from a diverse
range of natural images, which are used to refine the model. Our method
outperforms existing benchmarks in visual fidelity and lighting coherence,
showing superior generalization in real-world testing scenarios, highlighting
its versatility and practicality.
