  Current 3D content generation approaches build on diffusion models that
output RGB images. Modern graphics pipelines, however, require physically-based
rendering (PBR) material properties. We propose to model the PBR image
distribution directly, avoiding photometric inaccuracies in RGB generation and
the inherent ambiguity in extracting PBR from RGB. Existing paradigms for
cross-modal fine-tuning are not suited for PBR generation due to both a lack of
data and the high dimensionality of the output modalities: we overcome both
challenges by retaining a frozen RGB model and tightly linking a newly trained
PBR model using a novel cross-network communication paradigm. As the base RGB
model is fully frozen, the proposed method does not risk catastrophic
forgetting during fine-tuning and remains compatible with techniques such as
IPAdapter pretrained for the base RGB model. We validate our design choices,
robustness to data sparsity, and compare against existing paradigms with an
extensive experimental section.
