\section{Introduction}
\label{sec:intro}
\input{sections/intro.tex}

\section{Related Work}
\label{sec:rel}
\input{sections/rel.tex}

\section{Method}
\label{sec:method}
\input{sections/method.tex}

\input{sections/exp.tex}

\section{Limitations.}
\vspace{-0.5em}
\label{sec:slimit}
\input{sections/limitations.tex}

\vspace{-0.3em}
\section{Conclusion}
\vspace{-0.5em}
\label{sec:conclu}
\input{sections/conclu.tex}


{\small
\bibliographystyle{ieee_fullname}

\subsection{Sparse Volumes for Radiance Fields}
\label{sec:representation}



Our output radiance field is modeled by a sparse neural volume $\mathcal{V}$ that has per-voxel neural features in voxels that approximately cover the actual scene surface. 
We regress volume density $\sigma$ and view-dependent radiance $c$ at any given 3D location $x$ from this volume using an MLP network, in which we first tri-linearly sample a feature vector and then use the MLP to convert the feature to volume properties, expressed by 
\begin{align}
\label{eq:radiancefield}
\sigma, c &= R(x, d, \mathcal{V}(x)).
\end{align}
Here $\mathcal{V}(x)$ represents the trilinearly interpolated feature at $x$, $R$ is the MLP, and $d$ is the viewing direction in rendering. 
The output volume properties regressed from the volume can be directly used to synthesize images at novel target viewpoints via differentiable ray marching as is done in NeRF \cite{mildenhall2020nerf}. 
This radiance field representation is similar to the one in Neural Sparse Voxel Fields \cite{liu2020neural} that purely relies on per-scene optimization for the reconstruction. 
We instead propose to leverage neural networks trained across scenes to predict the neural volumes from image sequences.

In our pipeline, such sparse volumes are reconstructed locally as $\mathcal{V}_t$ per frame $t$ and also globally as $\mathcal{V}_g$ for the entire sequence. The MLP network $R$ is shared across all volumes in the training process. We model the local volumes and the global volume both in the canonical world space. 
 




\subsection{Reconstructing Local Volumes}
\label{sec:localrecon}
We propose a deep neural network to regress a local neural volume for each input frame $t$, using its image $I_t$ and $K-1$ images from neighboring views. Usually, given a monocular video, these neighboring views correspond to temporal neighboring frames. 
Using multiple nearby images for per-frame reconstruction allows the network to leverage multi-view correspondence to recover better scene geometry, which a single image cannot provide.

To make the local reconstruction per frame well generalized across scenes, we leverage deep MVS techniques \cite{yao2018mvsnet,ji2017surfacenet}, which are known to be generalizable. 
We extract 2D image features, build a cost volume from the features, and regress a neural feature volume from the cost volume.
However, unlike MVSNeRF \cite{chen2021mvsnerf} and other MVS techniques \cite{yao2018mvsnet,cheng2020deep} that built frustum volumes in view's perspective coordinate, we construct volumes in the canonical world coordinate frame to align it with the final global volume output $\mathcal{V}_g$, facilitating the following fusion process.
\vspace{-0.5em}



\boldstartspace{Image feature extraction.} 
We use a deep 2D convolutional neural network to extract 2D image features for each input image. This network maps the input image $I_t$ into a 2D feature map $F_t$, encoding the scene content from each view.
\vspace{-1.5em}

\boldstartspace{Local sparse volume.} 
We consider the bounding box that covers the frustums of all $K$ neighboring viewpoints in the world coordinate frame, containing of a set of voxels in the canonical space.
The bounding volume is axis-aligned with the world frame; each voxel inside it can be visible to a different number of neighboring views. We mask out all the voxels invisible to all view, leading to a sparse set of voxels in the bounding box.
We then unproject the image features into this volume for our local reconstruction.
\vspace{-0.3em}

\boldstartspace{3D feature volume.} 
For each neighboring viewpoint $i$ and its feature map $F_i$, we build a 3D feature volume $\mathcal{U}_i$.
In particular, for each visible voxel centered at $v$, we fetch the 2D image feature at its 2D projection from each neighboring view at frame $t$. In addition to pure image features, we leverage the corresponding viewing direction $d_i$ at $v$ from each viewpoint and compute additional features using an MLP $G$.
The per-view 3D volume $\mathcal{U}_i$ is expressed by
\begin{align}
\mathcal{U}_i(v) = [F_i(u_i), G(d_i)],
\end{align}
where $\mathcal{U}_i(v)$ is the feature at a voxel centered at $v$, $u_i$ is the center's 2D projection in view $i$, $[\cdot, \cdot]$ represents feature concatenation.
Note that, we encode the additional information of input viewing directions in the reconstruction process; this crucial information makes our following fusion module effectively account for the view-dependent effects captured across frames.
\vspace{-0.6em}

\boldstartspace{Neural reconstruction.} 
We then aggregate the features across multiple neighboring viewpoints to regress a local volume $\mathcal{V}_t$ at frame $t$, expressing a local radiance field.
We propose to leverage the mean and variance of the per-voxel features in $\mathcal{U}_i$ computed across neighboring viewpoints; such operations have been widely used in building cost volumes in MVS-based techniques \cite{yao2018mvsnet,chen2021mvsnerf}, where the mean can fuse per-view appearance information and the variance provides rich correspondence cues for geometry reasoning. 
These two operation are also invariant to the number/order of input; in our case, this naturally handles voxels that have different numbers of visible viewpoints.
We use a deep neural network $J$ to process the mean and variance features per voxel to regress the per-view reconstruction by
\begin{align}
\mathcal{V}_t &= J([\text{Mean}_{i \in \mathcal{N}_t}\mathcal{U}_i,\ \text{Var}_{i \in \mathcal{N}_t}\mathcal{U}_i]),
\end{align}
Here $\mathcal{N}_t$ represents all $K$ neighboring viewpoints used at frame $t$; $\text{Mean}$ and $\text{Var}$ represent element-wise average and variance operation, respectively.

Essentially, we regress the local radiance field from the features across neighboring views. This is similar to MVSNeRF \cite{chen2021mvsnerf}. However, unlike MVSNeRF that considers only local reconstruction and builds perspective frustrum volumes for small-baseline rendering, we leverage these local volumes for global large-scale reconstruction and rendering. We build volumes directly in canonical space, naturally providing per-frame voxel inputs for our fusion module.






\begin{figure}\centering
    \includegraphics[width=0.42\textwidth]{figures/ops.pdf}
    \captionof{figure}{A 2D example illustrating the GRU fusion step. The hidden state, which is also the global feature volume $\mathcal{V}^g_{t-1}$, is adaptively updated by aggregating new information in the incoming local feature volume $\mathcal{V}_t$.}
    \vspace{-1.4em}
    \label{fig:ops}
\end{figure}


\subsection{Fusing Volumes for Global Reconstruction}
\label{sec:fusion}

In order to create a consistent, efficient, and extensible scene reconstruction, we propose to use a global neural volume fusion network to incrementally fuse local feature volumes $\{\mathcal{V}_t\}$ per frame into a global volume $\mathcal{V}^g$. 

\boldstartspace{Fusion.}
At each frame $t$, we consider its local sparse volume reconstruction $\mathcal{V}_t$ and the global reconstruction $\mathcal{V}^g_{t-1}$ from the previous frame as recurrent input. 
We leverage GRUs (Gated Recurrent Unit)~\cite{cho2014learning} with sparse 3D CNNs in our fusion module, allowing our network to learn to recurrently fuse the per-frame local reconstruction and output high-quality global radiance fields.
This is expressed by
\begin{align}
    z_t &= M_z([\mathcal{V}^g_{t-1}, \mathcal{V}_t]), \\
    r_t &= M_r([\mathcal{V}^g_{t-1}, \mathcal{V}_t]), \\
    \tilde{\mathcal{V}}^g_{t} &= M_t([r_t * \mathcal{V}^g_{t-1}, \mathcal{V}_t]), \\
    \mathcal{V}^g_{t} &= (1-z_t) * \mathcal{V}^g_{t-1} + z_t * \tilde{\mathcal{V}}^g_{t},
\end{align}
where $*$ is the element-wise multiplication, $z_t$ and $r_t$ are the update gate and the reset gate, $M_z$, $M_r$ and $M_t$ all deep neural networks with sparse 3D convolution layers. As in standard GRU, $M_z$ and $M_r$ are designed with sigmoid activation in the end, while $M_t$ uses tanh, allowing for the entire model sequentially updating the global reconstruction $\mathcal{V}^g_{t}$ (seen as the hidden state in a GRU) for every input frame.
In this process, we only apply the networks on the voxels covered by the local volume $\mathcal{V}_t$; all other voxels in the global volume are kept unchanged. A 2D illustration of this GRU fusion process is shown in Fig.~\ref{fig:ops}.

Intuitively, the update gate $z_t$ and reset gate $r_t$ in the GRU determine how much information from the previous global volume $\mathcal{V}^g_{t-1}$ as well as how much information from the current local volume $\mathcal{V}_t$ should be incorporated into the new global features. In this way, our module can adaptively improve the global scene reconstruction by filling up holes and refining features while keeping the representation consistent. This fusion process is similar to previous 3D reconstruction pipelines~\cite{pizzoli2014remode,kar2017learning,sun2021neucon} that focus on geometry reconstruction; in contrast, we instead reconstruct neural feature volumes to represent neural radiance fields for volume rendering, leading to photo-realistic novel view synthesis.
\vspace{-0.5em}

\boldstartspace{Voxel pruning.}  
To maximize the memory and rendering efficiency, we adaptively prune the global volume reconstruction $\mathcal{V}^g_t$ for every frame by removing the non-essential voxels that do not have any scene content inside. We naturally leverage the volume density in each voxel regressed by our radiance field (Eqn.~\ref{eq:radiancefield}), which models the scene geometry. In particular, we prune voxels $V$ if:

\begin{equation}
    \min_{i=1...k} \exp(-\sigma(v_i)) > \gamma, v_i \in V, \label{eqn:prune}
\end{equation}

where $\{v_i\}_{i=1}^k$ are $k$ uniformly sampled points inside the voxel $V$, $\sigma(v_i)$ is the predicted density at location $v_i$, and $\gamma$ is a pruning threshold. This pruning step is performed in both later training phase and inference phase once we get a global feature volume $\mathcal{V}^g_t$. 
By doing so, we make our global volume sparser, leading to more efficient reconstruction and rendering.






\subsection{Training and optimization}
\label{sec:training}




Once a radiance field (that is represented by a sparse neural volume as described in Sec.~\ref{sec:representation}) is reconstructed, our final rendering is achieved via differentiable ray marching using the regressed volume density and view-dependent radiance at any sampled ray points, as is done in NeRF and any other radiance field methods \cite{mildenhall2020nerf,liu2020neural}. 
In this work, our full pipeline is trained, completely depending on the rendering supervision with the ground truth images, without any extra geometry supervision.

In particular, we first train our local reconstruction network and the radiance field decoder (R) with a loss 
\begin{equation}
    \mathcal{L}_\text{local} = \|C_t -\hat{C} \|^2_2, \label{eq:losslocal}
\end{equation}
where $\hat{C}$ is the ground truth pixel color and $C_t$ represents the rendered pixel color using the local volume $\mathcal{V}_t$ reconstructed at frame $t$.
This makes the network learn to predict reasonable local neural volumes, which are already renderable and able to produce realistic images locally; it also initializes the radiance field decoder MLP to a reasonable state, which is later shared across local and global volumes.
This pre-training allows the local reconstruction module to provide meaningful volume features for the fusion module to utilize in the end-to-end training, effectively facilitating the fusion task.
We then train our full pipeline with the local reconstruction network, fusion network, and the radiance field decoder network all together from end to end, using a rendering loss:
\begin{equation}
    \mathcal{L}_\text{fuse} = \sum_t \|C_t - \hat{C} \|^2_2 + \|C^g_t - \hat{C} \|^2_2 ,
\end{equation}
where $C_t$ is the pixel color rendered from the local reconstruction $\mathcal{V}_t$ (as is in Eqn.~\ref{eq:losslocal}) and $C^g_t$ is the color rendered from the global volume $\mathcal{V}^g_t$ after fusing frame $t$.  
Basically, we take every intermediate global and local volume ($\mathcal{V}_t$ and $\mathcal{V}^g_t$) at every frame to render novel view images and supervise them with the ground truth.
The fusion module thus reasonably learns to fuse local volumes from an arbitrary number of input frames.

After trained, our full network is able to output a high-quality radiance field from direct network inference and produce realistic rendering results (as shown in Fig.~\ref{fig:teaser}).
In addition, the reconstructed radiance field as a sparse neural volume can also be easily optimized (fine-tuned) per scene further to boost the rendering quality. 
\vspace{-0.5em}



\boldstartspace{Fine-tuning.}
To fine-tune the estimated radiance field, we optimize the per-voxel neural features in the sparse volume reconstruction $\mathcal{V}^g$ and the MLP decoder per scene with the captured images, leading to better rendering results.
Since our initial reconstruction is already very good, a short period of optimization with less than 25k iterations can usually lead to very high quality, which takes less than 1 hour.
This is substantially less optimization time than NeRF and other pure per-scene optimization methods.

In this per-scene optimization stage, we also do a coarse-to-fine reconstruction, similar to NSVF \cite{liu2020neural}.
Basically, after every 10k optimization iterations, we further prune unnecessary voxels (using Eqn.~\ref{eqn:prune}) and also subdivide each voxel into 8 sub voxels. This prune and subdivision step progressively increases the spatial resolution of the neural volume, further improving our final rendering quality.








\section{Implementation Details.}
\label{sec:impl}

\boldstart{Training datasets.} 
Our training data consists of both large-scale indoor scenes from ScanNet~\cite{dai2017scannet} and small objects from DTU~\cite{jensen2014large} and Google Scanned Objects~\cite{scannedobjects}. 
We randomly sample 100 scenes from ScanNet for training. 
For DTU, We adopt the training split from PixelNeRF~\cite{yu2020pixelnerf}, which includes 88 training scenes. 
In addition, we use the object-centric synthetic renderings of 1,023 models from the Google Scanned Objects~\cite{scannedobjects} generated by~\cite{ibrnet}. 
Our training data includes various camera setups and scene types, enabling our model to generalize to all kinds of scenarios.
We demonstrate that our model can effectively work on large-scale indoor scenes, as well as scenes of objects.

\boldstartspace{Training details.} For each input image sequence, we uniformly sample key frames from the full sequences for the input frames in network training. For object-centric datasets, we sample 16 views for each scene, and for ScanNet, we sample $2\% - 5\%$ of the full sequence as key frames. All other frames are used for supervision.
We use $K=3$ neighboring views for each input frame, for local volume reconstruction. 
For video sequence captured by a monocular camera, such as the scenes in ScanNet, we directly take the 3 neighboring key frames temporally. For other datasets, we select the 3 spatially closest viewpoints, in terms of both viewing location and direction.

The sparse volumes and networks are implemented with torchsparse~\cite{tang2020searching}. We train our model using Adam optimizer with an initial learning rate of $0.003$. We train our network with 2 NVIDIA 2080Ti GPUs for 3 days.
During inference,
our network processes frames from ScanNet sequences in real-time at 22 FPS. The final model takes 38 seconds on average to render a $640 \times 480$ image on ScanNet . 


\section{Results}
\label{sec:exp}

In this section, we evaluate the our model on various datasets.
For all results, we denote our results from direct network inference as Ours and our results after per-scene fine-tuning as Ours\textsubscript{ft} in all figure and tables. Similar labels are applied to IBRNet and other generalizing methods.
\vspace{-0.5em}



\boldstartspace{Baselines.}
We compare our method against the state-of-the-art NeRF methods on novel view synthesis including per-scene optimization methods, such as NeRF~\cite{mildenhall2020nerf}, NVSF~\cite{liu2020neural}, and NerfingMVS~\cite{wei2021nerfingmvs}, and methods that can generalize to new scenes, such as PixelNeRF~\cite{yu2020pixelnerf}, IBRNet~\cite{ibrnet}, and MVSNeRF~\cite{chen2021mvsnerf}.


To achieve fair and accurate comparisons, we run our method on the same experiment settings in previous papers, and we try our best to directly use the reported official quantitative results in previous papers or use the official code to run the experiments.
We find that the official NeRF and IBRNet code can easily run and work on different datasets, producing corresponding images.
We demonstrate visual comparison with these two methods across the three testing sets in Fig.~\ref{fig:results}.
On the other hand, NSVF is very hard to run without enough GPU memory; their official models are optimized on a V100 GPU that has 32G memory; we found it impractical to generate their corresponding results with our resources. We therefore only include NSVF's quantitative results whenever they are reported previously.
Besides, the recent MVSNeRF~\cite{chen2021mvsnerf} is a very relevant technique, but it is designed to take a fixed number of three nearby views as its network input; as a result, it cannot support large-baseline rendering or arbitrary number of input images for inference. We therefore only compare with MVSNeRF on the DTU dataset in the same experiment setting used in their paper.
\vspace{-0.8em}

\begin{table}
    \centering
    \setlength{\tabcolsep}{4.0pt}
    \begin{tabular}{@{}lcccc@{}}
        \toprule
        Method      & Settings & PSNR$\uparrow$ & SSIM$\uparrow$ & LPIPS$\downarrow$  \\
        \midrule
        IBRNet                          & \multirow{2}{*}{\shortstack{No per-scene \\ optimization}}                  &{21.19}      &{0.786}       &{0.358}          \\
        Ours                            &              &\mathcal{B}{{22.99}}  &\mathcal{B}{{0.838}}   &\mathcal{B}{{0.335}}     \\
        \midrule
        NeRF                            &  \multirow{5}{*}{\shortstack{Per-scene \\ optimization}}              &24.04        &0.860         &0.334    \\
        NSVF                            &            &26.01        &0.881         &-         \\
        NeRFingMVS                      &            &26.37        &0.903         &0.245     \\
        IBRNet\textsubscript{ft-1.5h}   &            &{25.14}      &{0.871}       &{0.266}    \\
        Ours\textsubscript{ft-1h}       &            &\mathcal{B}{{26.49}}  &\mathcal{B}{{0.915}}   &\mathcal{B}{{0.209}}   \\
        \bottomrule
    \end{tabular}\vspace{-0.5em}
   
    \caption{
        Quantitative comparisons on the ScanNet dataset~\cite{dai2017scannet}. We follow the same experiment settings 
        as in NeRFingMVS~\cite{wei2021nerfingmvs} and report the error metrics including PSNR (higher is better), SSIM (higher is better) and 
        LPIPS (lower is better). 
        Note that, our diret inference results are better than IBRNet~\cite{ibrnet}.
        Our fine-tuning results achieve the best numbers in all three metrics.
       
       
       
       
    }
    \label{tab:scannet8}
\end{table}




\begin{table}
    \centering
    \setlength{\tabcolsep}{4.0pt}
    \begin{tabular}{@{}lcccc@{}}
        \toprule
        Method      & Settings & PSNR$\uparrow$ & SSIM$\uparrow$ & LPIPS$\downarrow$      \\
        \midrule
        IBRNet          & \multirow{2}{*}{\shortstack{No per-scene \\ optimization}}                                 &\mathcal{B}{25.51}  &0.916        &0.100         \\
        Ours       &                                    &\mathcal{B}{{25.47}}    &\mathcal{B}{{0.922}}  &\mathcal{B}{{0.093}}   \\
        \midrule
        NeRF         & \multirow{4}{*}{\shortstack{Per-scene \\ optimization}}                                    &31.01      &0.947        &0.081       \\
        NSVF         &                                 &\mathcal{B}{31.75}  &\mathcal{B}{0.954}    &\mathcal{B}{0.048}   \\
        IBRNet\textsubscript{ft-1.5h}  &                &28.19      &0.943        &0.072       \\
        Ours\textsubscript{ft-1h}  &               &{31.25}    &\mathcal{B}{{0.953}}  &{0.069}     \\
        \bottomrule
    \end{tabular}\vspace{-0.5em}
   
    \caption{
        Quantitative comparison on the NeRF Synthetic dataset~\cite{mildenhall2020nerf}.
       
        Our model is able to generate better results than IBRNet in both direct inference 
        and fine-tuning settings. Our model after 1 hour fine-tuning achieves comparable performance
        to the state-of-the-art per-scene overfitting methods such as NeRF~\cite{mildenhall2020nerf}
        and NVSF~\cite{liu2020neural}. 
    }\vspace{-1.2em}
    \label{tab:nerfsynth}
\end{table}


\boldstartspace{Large-scale scenes in ScanNet.} 
We follow the same training and evaluation scheme as described in
NerfingMVS~\cite{wei2021nerfingmvs} for the comparison on ScanNet. 
We tested our model on the $8$ testing scenes used in their paper\comment{, where each scene contains around 32 training and 8 testing views}.  
From Table~\ref{tab:scannet8}, we can see that our recurrent neural reconstruction network generates significantly
better results than IBRNet via direct network inference.
After fine-tuning for only a short period of 1 hour, the quality of our results is further boosted significantly, leading to the best PSNR, SSIM and LIPIPS in all compared methods. 
Note that, the per-scene optimization methods like NeRF, NeRFing MVS and NSVF require substantially longer per-scene optimization time but are still outperformed by our method.
Our approach is impressively better than NSVF in this case though both methods have similar final radiance field representation; this indicates that the data priors learned by our recurrent neural network can effectively help the reconstruction and lead to reasonable initial radiance fields, even benefiting the per-scene fine-tuning process.

As shown in Fig.~\ref{fig:results}, our results on these large-scale scenes are of very high visual quality.  Our results are visually much better than the IBRNet's results from both direct inference and per-scene fine-tuning. 
IBRNet generates tearing artifacts since it performs image-based rendering and
can only aggregate a small set of local neighboring views due to limited GPU
memory. In contrast, our model learns a unified 3D representation in the canonical space with a recurrent
module that is able to efficiently aggregate per-view information across all input views, leading to significantly better rendering quality with better across-view consistency.
Note that, even our direct inference renderings are already very realistic and contain few noticeable artifacts; they are arguably comparable to the rendering results of NeRF which require long per-scene optimization. 
Our approach achieves highly efficient and highly accurate large-scale radiance field reconstruction.
\vspace{-0.5em}

\begin{table}
    \centering
    \setlength{\tabcolsep}{2.5pt}
    \begin{tabular}{@{}lcccc@{}}
        \toprule
        Method      & Settings & PSNR$\uparrow$ & SSIM$\uparrow$ & LPIPS$\downarrow$  \\
        \midrule
        PixelNeRF       & \multirow{4}{*}{\shortstack{No per-scene \\ optimization}}                             &19.31         &0.789      &0.382    \\
        IBRNet &                                      &26.04         &0.917      &0.190   \\
        MVSNeRF &                                    &\mathcal{B}{26.63}     &\mathcal{B}{0.931}  &\mathcal{B}{0.168}   \\
        Ours     &                                   &{26.19}       &{0.922}    &{0.177}   \\
        \midrule
        NeRF            &  \multirow{4}{*}{\shortstack{Per-scene \\ optimization}}                              &27.01         &0.902      &0.263   \\
        IBRNet\textsubscript{ft-1.5h}       &        &31.35         &0.956      &0.131   \\
        MVSNeRF\textsubscript{ft-15min}      &       &28.50         &0.933      &0.179   \\
        Ours\textsubscript{ft-1h}             &      &\mathcal{B}{{31.79}}   &\mathcal{B}{0.962}  &\mathcal{B}{0.119}   \\
        \bottomrule
    \end{tabular}\vspace{-0.5em}
   
    \caption{
        Quantitative comparisons on the DTU dataset~\cite{dtu}. Our model is able to generate good results under this difficult setting where only 3 input views are given for the direct inference. Our fine-tuning results outperforms other methods in all three metrics.
    }\vspace{-1.2em}
    \label{tab:dtu3}
\end{table}

\begin{figure*}\centering
    \includegraphics[width=\textwidth]{figures/results-v2.pdf}
    \captionof{figure}{
        Qualitative comparisons of rendering quality on diverse scenes between our method and state-of-the-art 
        method. Our method achieves better performance than state-of-the-art 
        generalizable method IBRNet~\cite{ibrnet} in both the direct-inference 
        and the fine-tuned settings, where 
        IBRNet generates results with obvious blurry and tearing artifacts. Our model fine-tuned 
        for 1 hour can generate even better results than NeRF~\cite{mildenhall2020nerf} that requires 12 hours of training, especially 
        on large-scale scenes from ScanNet~\cite{dai2017scannet}.
    }
    \vspace{-1.5em}
    \label{fig:results}
\end{figure*}



\boldstartspace{NeRF Synthetic.} 
Our method also works well on small-scale scenes.
We conduct experiments on the NeRF Synthetic $360^{\circ}$ 
dataset, and apply the same evaluation setting as in~\cite{mildenhall2020nerf}. 
As shown in Table~\ref{tab:nerfsynth}, without per-scene fine-tuning, our model generates results that are 
comparable to IBRNet; however, fine-tuning significantly boosts the performance 
of our model, leading to high accuracy that is much superior to the fine-tuned IBRNet. In practice, IBRNet 
suffers from the sparsely distributed input views of the dataset with large baselines, where 
interpolating neighboring views are not effective to synthesize realistic novel view images. 
Our fine-tuned 
model also achieves similar performance when compared to per-scene optimization 
methods \cite{mildenhall2020nerf,liu2020neural}, while ours is optimized for only 1 hour, substantially less than the time other methods require.
\vspace{-0.5em}

\boldstartspace{DTU.} To show that our method works with a small number of input views with small baselines. We also evaluate our model on the DTU dataset, following the experiment settings in MVSNeRF \cite{chen2021mvsnerf}, where only 3 views are provided for the setting without per-scene optimization and 16 more views are provided in the per-scene optimization setting. Here we also compare with PixelNeRF \cite{yu2020pixelnerf}, which is specifically trained for the DTU dataset in their paper.
As demonstrated in Tab.~\ref{tab:dtu3} and Fig.~\ref{fig:results}, similar to previous results, 
our model generalizes well to the testing scenes and can be efficiently fine-tuned to 
outperform NeRF and other generalizable methods including IBRNet~\cite{ibrnet} and MVSNeRF~\cite{chen2021mvsnerf}. 
















\section{Additional Details}
\label{sec:smodel}

We follow~\cite{sun2021neucon} to use a modified MnasNet~\cite{tan2019mnasnet} pretrained from ImageNet~\cite{russakovsky2015imagenet} as the 2D encoder. The output feature channel number from the 2D encoder is 64. The direction encoder $G$ is a 5 layer MLP with 16-channel output. The SparseConvNet $J$ has 5 SparseConv layers implemented with torchsparse~\cite{tang2020searching}. Each of $M_z, M_r, M_t$ has 3 SparseConv layers. All networks use ReLU as activation layers. All features in the feature volumes (global $\mathcal{V}^g_t$ or local $\mathcal{V}_t$) have 16 channels. We apply positional encoding to the volume feature with maximum frequency $L=5$ before feeding them into the volume renderer. The initial voxel size is set manually for different datasets, specifically $40mm$ for large-scale datasets \eg ScanNet~\cite{dai2017scannet}, and $4mm$ for object-centric datasets \eg NeRF Synthetic~\cite{mildenhall2020nerf} and Google Scanned Objects~\cite{scannedobjects}. When unprojecting 2D features into local feature volumes, we build view frustums with max depth $d_{\max} = 3m$. The pruning threshold $\gamma$ is set to $0.6$ for all experiments. Nearest-neighbor interpolation is used for all coarse-to-fine fine-tuning experiments. The code will be released after the review period.

\section{Additional Results}
\label{sec:sresults}

\boldstart{Effect of input view numbers.} Figure~\ref{fig:nviews} shows the comparison when sample different numbers of neighboring views to build the feature volume. Our learned global feature fusion module is able to effectively fuse information from different views. Note that, when using more input views, our method could produce sharper details in the rendered images, which are very close to the reference. 

\boldstartspace{Effect of fine-tuning.} Figure~\ref{fig:finetune} shows the visual quality of our method when fine-tuning for different time period. Note that our direct inference result have outperformed NeRF, and the results from fine-tuning contain significantly more details. PSNRs are shown at top right of each image.

\boldstartspace{Geometry reconstruction.} Following~\cite{mildenhall2020nerf,chen2021mvsnerf}, we evaluate our geometry reconstruction quality on the DTU dataset~\cite{jensen2014large} by comparing depth reconstruction results generated from the volume density by a weighted sum of the depth values of the sampled points on marched rays. We use 16 input views and compare the depth quality on both input views and novel views with 2 common metrics. The results are shown in Table~\ref{tab:geo}. Thanks to the explicit geometry modeling in our pipeline, our approach achieves significantly more accurate geometry than other neural rendering methods.

\begin{figure}\centering
    \includegraphics[width=0.47\textwidth]{figures/nviews.pdf}
    \captionof{figure}{Effect of input view numbers. We show the results when sample different numbers of neighboring views to build the feature volume. All results are from the pretrained model without any further fine-tuning. 
   
    }
    \label{fig:nviews}\vspace{-0.6em}
\end{figure}

\begin{table}[t]
	\centering
	\begin{tabular}{lcc}
		\toprule
		Method & Abs Err$\downarrow$ & Acc (8mm)$\uparrow$ \\
		\midrule
		PixelNeRF~\cite{yu2020pixelnerf}   &  $0.205/0.211$ & $0.096/0.089$ \\
		IBRNet~\cite{ibrnet}    &  $ 1.123 / 1.324$ & $ 0.000 /0.000$\\
		Ours       &  $\textbf{0.034}/\textbf{0.036}$ & $\textbf{0.722}/\textbf{0.709}$\\
		\bottomrule
\end{tabular}
\rule{0pt}{1.0pt}
\caption{\textbf{Geometry reconstruction.} We evaluate depth reconstruction on the DTU testing set and compare with other two neural rendering methods PixelNeRF~\cite{yu2020pixelnerf} and IBRNet~\cite{ibrnet}. Our method significantly outperforms other neural rendering methods and achieves high depth accuracy. The two numbers of each item refers to the depth at \textit{input} / \textit{novel} views. }
\label{tab:geo}
\end{table}

















