  Recent advancements in controllable human image generation have led to
zero-shot generation using structural signals (e.g., pose, depth) or facial
appearance. Yet, generating human images conditioned on multiple parts of human
appearance remains challenging. Addressing this, we introduce Parts2Whole, a
novel framework designed for generating customized portraits from multiple
reference images, including pose images and various aspects of human
appearance. To achieve this, we first develop a semantic-aware appearance
encoder to retain details of different human parts, which processes each image
based on its textual label to a series of multi-scale feature maps rather than
one image token, preserving the image dimension. Second, our framework supports
multi-image conditioned generation through a shared self-attention mechanism
that operates across reference and target features during the diffusion
process. We enhance the vanilla attention mechanism by incorporating mask
information from the reference human images, allowing for the precise selection
of any part. Extensive experiments demonstrate the superiority of our approach
over existing alternatives, offering advanced capabilities for multi-part
controllable human image customization. See our project page at
https://huanngzh.github.io/Parts2Whole/.
