  In this paper, we focus on the One-shot Novel View Synthesis (O-NVS) task
which targets synthesizing photo-realistic novel views given only one reference
image per scene. Previous One-shot Generalizable Neural Radiance Fields
(OG-NeRF) methods solve this task in an inference-time finetuning-free manner,
yet suffer the blurry issue due to the encoder-only architecture that highly
relies on the limited reference image. On the other hand, recent
diffusion-based image-to-3d methods show vivid plausible results via distilling
pre-trained 2D diffusion models into a 3D representation, yet require tedious
per-scene optimization. Targeting these issues, we propose the GD$^2$-NeRF, a
Generative Detail compensation framework via GAN and Diffusion that is both
inference-time finetuning-free and with vivid plausible details. In detail,
following a coarse-to-fine strategy, GD$^2$-NeRF is mainly composed of a
One-stage Parallel Pipeline (OPP) and a 3D-consistent Detail Enhancer
(Diff3DE). At the coarse stage, OPP first efficiently inserts the GAN model
into the existing OG-NeRF pipeline for primarily relieving the blurry issue
with in-distribution priors captured from the training dataset, achieving a
good balance between sharpness (LPIPS, FID) and fidelity (PSNR, SSIM). Then, at
the fine stage, Diff3DE further leverages the pre-trained image diffusion
models to complement rich out-distribution details while maintaining decent 3D
consistency. Extensive experiments on both the synthetic and real-world
datasets show that GD$^2$-NeRF noticeably improves the details while without
per-scene finetuning.
