Outdoor inverse rendering from a single image using multiview self-supervision
Ye~Yu% \thanks{Y. Yu and W. Smith are with the Department of Computer Science
University of York
York
UK. Email: \{yy1571
william.smith\
In this paper we show how to perform scene-level inverse rendering to recover shape, reflectance and lighting from a single, uncontrolled image using a fully convolutional neural network. The network takes an RGB image as input, regresses albedo, shadow and normal maps from which we infer least squares optimal spherical harmonic lighting coefficients. Our network is trained using large uncontrolled multiview and timelapse image collections without ground truth. By incorporating a differentiable renderer, our network can learn from self-supervision. Since the problem is ill-posed we introduce additional supervision. Our key insight is to perform offline multiview stereo (MVS) on images containing rich illumination variation. From the MVS pose and depth maps, we can cross project between overlapping views such that Siamese training can be used to ensure consistent estimation of photometric invariants. MVS depth also provides direct coarse supervision for normal map estimation. We believe this is the first attempt to use MVS supervision for learning inverse rendering. In addition, we learn a statistical natural illumination prior. We evaluate performance on inverse rendering, normal map estimation and intrinsic image decomposition benchmarks.
main.tex
\documentclass[10pt,journal,compsoc,twoside]{IEEEtran}
\pdfoutput=1

\usepackage[pdftex]{graphicx}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{adjustbox}
% \usepackage{subfig}
\usepackage{multicol,bm,cite,multirow,array}
\usepackage[bookmarks=false]{hyperref}
\usepackage{color}

\newcommand{\etal}{{et al.~}}
\newcommand{\ie}{{i.e.~}}
\newcommand\mypara[1]{\vspace{0.2cm}\noindent \textbf{#1} \hspace{0.2cm}}

\captionsetup[subfigure]{skip=0pt}

\newcommand{\R}{{\mathbb{R}}}


\begin{document}

\title{Outdoor inverse rendering from a single image using multiview self-supervision}

\author{Ye~Yu%
\thanks{Y. Yu and W. Smith are with the Department
of Computer Science, University of York, York,
UK. Email: \{yy1571,william.smith\}@york.ac.uk}
and~William~A.~P.~Smith% <-this % stops a space
}

% The paper headers
\markboth{IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,~Vol.~X, No.~X, January~2021}%
{Yu and Smith: Outdoor inverse rendering from a single image using multiview self-supervision}

\IEEEtitleabstractindextext{%
%%%%%%%%% ABSTRACT
\begin{abstract}
In this paper we show how to perform scene-level inverse rendering to recover shape, reflectance and lighting from a single, uncontrolled image using a fully convolutional neural network. The network takes an RGB image as input, regresses albedo, shadow and normal maps from which we infer least squares optimal spherical harmonic lighting coefficients. Our network is trained using large uncontrolled multiview and timelapse image collections without ground truth. By incorporating a differentiable renderer, our network can learn from self-supervision. Since the problem is ill-posed we introduce additional supervision. Our key insight is to perform offline multiview stereo (MVS) on images containing rich illumination variation. From the MVS pose and depth maps, we can cross project between overlapping views such that Siamese training can be used to ensure consistent estimation of photometric invariants. MVS depth also provides direct coarse supervision for normal map estimation. We believe this is the first attempt to use MVS supervision for learning inverse rendering. In addition, we learn a statistical natural illumination prior. We evaluate performance on inverse rendering, normal map estimation and intrinsic image decomposition benchmarks.
\end{abstract}

% Note that keywords are not normally used for peerreview papers.
\begin{IEEEkeywords}
inverse rendering, shape-from-shading, intrinsic image decomposition, illumination estimation
\end{IEEEkeywords}}

% make the title area
\maketitle


% To allow for easy dual compilation without having to reenter the
% abstract/keywords data, the \IEEEtitleabstractindextext text will
% not be used in maketitle, but will appear (i.e., to be "transported")
% here as \IEEEdisplaynontitleabstractindextext when the compsoc 
% or transmag modes are not selected <OR> if conference mode is selected 
% - because all conference papers position the abstract like regular
% papers do.
\IEEEdisplaynontitleabstractindextext
% \IEEEdisplaynontitleabstractindextext has no effect when using
% compsoc or transmag under a non-conference mode.



% For peer review papers, you can put extra information on the cover
% page as needed:
% \ifCLASSOPTIONpeerreview
% \begin{center} \bfseries EDICS Category: 3-BBND \end{center}
% \fi
%
% For peerreview papers, this IEEEtran command inserts a page break and
% creates the second title. It will be ignored for other modes.
\IEEEpeerreviewmaketitle


\begin{figure*}[!t]
    \centering
    % \includegraphics[width=\textwidth,height=4cm]{example-image}
        \begingroup
\setlength{\tabcolsep}{1pt}
\renewcommand{\arraystretch}{0.5}
\resizebox{\textwidth}{!}{
\begin{tabular}{ccccccc}
\includegraphics[width=2.4cm]{teasers/teaser2/img.png}&
\includegraphics[width=2.4cm]{teasers/teaser2/albedo.png}&
\includegraphics[width=2.4cm]{teasers/teaser2/shadow.png}&
\includegraphics[width=2.4cm]{teasers/teaser2/nm_pred.png}&
\includegraphics[width=2.4cm]{teasers/teaser2/lighting.png}&
\includegraphics[width=2.4cm]{teasers/teaser2/geo_pred.png}&
\includegraphics[width=2.4cm]{teasers/teaser2/shading.png}
\\
\includegraphics[width=2.4cm]{teasers/teaser4/img_sky.png}&
\includegraphics[width=2.4cm]{teasers/teaser4/albedo.png}&
\includegraphics[width=2.4cm]{teasers/teaser4/shadow.png}&
\includegraphics[width=2.4cm]{teasers/teaser4/nm_pred.png}&
\includegraphics[width=2.4cm]{teasers/teaser4/lighting.png}&
\includegraphics[width=2.4cm]{teasers/teaser4/geo_pred.png}&
\includegraphics[width=2.4cm]{teasers/teaser4/shading.png}
\\
\includegraphics[width=2.4cm]{teasers/teaser6/img_sky.png}&
\includegraphics[width=2.4cm]{teasers/teaser6/albedo.png}&
\includegraphics[width=2.4cm]{teasers/teaser6/shadow.png}&
\includegraphics[width=2.4cm]{teasers/teaser6/nm_pred.png}&
\includegraphics[width=2.4cm]{teasers/teaser6/lighting.png}&
\includegraphics[width=2.4cm]{teasers/teaser6/geo_pred.png}&
\includegraphics[width=2.4cm]{teasers/teaser6/shading.png}
\\
\includegraphics[width=2.4cm]{teasers/teaser3/img_sky.png}&
\includegraphics[width=2.4cm]{teasers/teaser3/albedo.png}&
\includegraphics[width=2.4cm]{teasers/teaser3/shadow.png}&
\includegraphics[width=2.4cm]{teasers/teaser3/nm_pred.png}&
\includegraphics[width=2.4cm]{teasers/teaser3/lighting.png}&
\includegraphics[width=2.4cm]{teasers/teaser3/geo_pred.png}&
\includegraphics[width=2.4cm]{teasers/teaser3/shading.png}
\\
Input & Diffuse albedo & Shadow & Normal map & Illumination & Frontal shading & Shading \\
\end{tabular}
}
\endgroup
    \caption{Sample output from our inverse rendering network. From a single image (col.~1), we estimate albedo, shadow and normal maps and illumination (col.~2-5); re-rendering of our shape with (col.~6) frontal, white point source and with (col.~7) estimated spherical harmonic lighting.}
    \label{fig:teaser}
    \vspace{-0.3cm}
\end{figure*}



\IEEEraisesectionheading{\section{Introduction}}\label{sec:intro}

\IEEEPARstart{I}{nverse} rendering is the problem of estimating one or more of illumination, reflectance properties and shape from observed appearance (i.e.~one or more images). In this paper, we tackle the most challenging setting of this problem; we seek to estimate all three quantities from only a single, uncontrolled image of an arbitrary scene. Specifically, we estimate a normal map, diffuse albedo map, cast shadow map and spherical harmonic lighting coefficients. This subsumes, and is more challenging than, several classical computer vision problems: (uncalibrated) shape-from-shading, intrinsic image decomposition, shadow removal and lighting estimation.

Classical approaches \cite{BarronTPAMI2015,Langguth:12} cast these problems in terms of energy minimisation. Here, a data term measures the difference between the input image and the synthesised image that arises from the estimated quantities. We approach the problem as one of image to image translation and solve it using a deep, fully convolutional neural network. 
However, there are no existing methods or devices that allow the capture of accurate ground truth for inverse rendering of uncontrolled, outdoor scenes. Hence, the labels required for supervised learning are not readily available.
Instead, we use the data term for self-supervision via a differentiable renderer (see Fig.~\ref{fig:inference}).

Single image inverse rendering is an inherently ambiguous problem. For example, any image can be explained with zero data error by setting the albedo map equal to the image, the normal map to be planar and the illumination such that the shading is unity everywhere. Hence, the data term alone cannot be used to solve this problem. For this reason, classical methods augment the data term with generic \cite{BarronTPAMI2015} or object-class-specific \cite{aldrian2013inverse} priors. Likewise, we also exploit a statistical prior on lighting. However, our key insight that enables the CNN to learn good performance is to introduce additional supervision provided by an offline multiview reconstruction or from time-lapse video.

While photometric vision has largely been confined to restrictive lab settings, classical geometric methods are sufficiently robust to provide multiview 3D shape reconstructions from large, unstructured datasets containing very rich illumination variation \cite{Snavely:08, Furukawa:10}. This is made possible by local image descriptors that are largely invariant to illumination. However, these methods recover only geometric information and any recovered texture map has illumination ``baked in'' and so is useless for relighting.
We exploit the robustness of geometric methods to varying illumination to supervise our inverse rendering network. We apply a multiview stereo (MVS) pipeline to large sets of images of the same scene. We select pairs of overlapping images with different illumination, use the estimated relative pose and depth maps to cross project photometric invariants between views and use this for supervision via Siamese training. In other words, geometry provides correspondence that allows us to simulate varying illumination from a fixed viewpoint. Finally, the depth maps from MVS provide coarse normal map estimates that can be used for direct supervision of the normal map estimation. This is important for our training schedule but note that the subsequent self-supervised learning allows the network to perform significantly better normal map estimation than simply training a network in a supervised fashion with MVS normal maps. Time-lapse video provides another useful constraint since albedo and surface normal maps remain constant while lighting (and therefore shading and shadowing) vary.

\subsection{Contribution}

Deep learning has already shown good performance on components of the inverse rendering problem. This includes monocular depth estimation \cite{eigen2014depth}, depth and normal estimation \cite{eigen2015predicting} and intrinsic image decomposition \cite{lettry2018darn}. However, these works use supervised learning. For tasks where ground truth does not exist, such approaches must either train on synthetic data (in which case generalisation to the real world is not guaranteed) or generate pseudo ground truth using an existing method (in which case the network is just learning to replicate the performance of the existing method). 
In this context, we make the following contributions. 
To the best of our knowledge, we are the first to exploit MVS supervision for learning inverse rendering. Second, we are the first to tackle the most general version of the problem, considering arbitrary outdoor scenes and learning from real data, as opposed to restricting to a single object class \cite{tewari2017mofa} or using synthetic training data \cite{zheng2018t2net}. Third, we introduce a statistical model of spherical harmonic lighting in natural scenes that we use as a prior. Finally, the resulting network is the first to inverse render all of shape, reflectance and lighting in the wild, outdoors.

An earlier version of the work in this paper was originally presented in \cite{yu2019inverserendernet}. Here, we have extended the method in a number of ways. First, we additionally estimate a shadow map, avoiding cast shadows from being baked into the albedo map. Second, we improve the labels used for direct normal map supervision by explicitly detecting ground plane pixels (which have unreliable MVS depth estimates) and replacing them with an estimated ground plane normal direction. Third, we reformulate the least squares solution for lighting parameters to avoid numerical instability. Fourth, we introduce a new source of training data by exploiting geometry and reflectance invariants in fixed position time-lapse video. Fifth, we also make changes to the architecture, use perceptual instead of mean squared error losses and use high resolution cross projections to avoid blurring the supervision signal. Taken together, these extensions improve performance and mean that we avoid the need to use generic priors such as smoothness as used in \cite{yu2019inverserendernet}. Finally, we provide a more thorough evaluation.


\section{Related work}

\mypara{Classical approaches} 
Classical methods estimate intrinsic properties by fitting photometric or geometric models using optimisation.
Most methods require multiple images. From multiview images, a structure-from-motion/multiview stereo pipeline enables recovery of dense mesh models \cite{Kazhdan:13, Furukawa:10} though illumination effects are baked into the texture. From images with fixed viewpoint but varying illumination, photometric stereo can be applied. More sophisticated variants consider statistical BRDF models \cite{Alldrin:08}, the use of outdoor time-lapse images \cite{Langguth:12} and spatially-varying BRDFs \cite{Goldman:09}.
Attempts to combine geometric and photometric methods are limited. Haber \etal \cite{Haber:09} assume known geometry (which can be provided by MVS) and inverse render reflectance and lighting from community photo collections. Kim \etal \cite{kim2016multi} represents the state-of-the-art and again use an MVS initialisation for joint optimisation of geometry, illumination and albedo.
Some methods consider a single image setting.
Jeson \etal \cite{jeon2014intrinsic} introduce a local-adaptive reflectance smoothness constraint for intrinsic image decomposition on texture-free input images which are acquired with a texture separation algorithm. Barron \etal \cite{BarronTPAMI2015} present SIRFS, a classical optimisation-based approach that recovers all of shape, illumination and albedo using a sophisticated combination of generic priors. 
Romeiro \etal \cite{romeiro2008passive} simplify the 4D BRDF function to a bivariate representation by assuming an isotropic BRDF function is invariant to rotation about the half vector, and estimate the BRDF in a simple setting where illumination and geometry are known. 
Instead of tackling the problem in an optimisation-based manner, \cite{lombardi2015reflectance, oxholm2015shape, lombardi2016radiometric} formulate inverse rendering or sub-problems as a Bayesian probabilistic problem, paired with empirical and data-driven prior probabilities, and produce the solution by means of MAP estimation.
Romeiro \etal \cite{romeiro2010blind} exploit the statistical distributions of natural environment maps at different scales as the prior probability distribution for the unknown illumination, and find the optimal BRDF function given the geometry, input image and the illumination prior distributions.

\mypara{Deep depth prediction}
Direct estimation of shape alone using deep neural networks has attracted a lot of attention. Eigen \etal \cite{eigen2014depth, eigen2015predicting} were the first to apply deep learning in this context. Subsequently, performance gains were obtained using improved architectures \cite{laina2016deeper}, post-processing with classical CRF-based methods \cite{wang2015towards, liu2015deep, xu2017multi} and using ordinal relationships for objects within the scenes \cite{fu2018deep, MegaDepthLi18, chen2016single}. Zheng \etal \cite{zheng2018t2net} use synthetic images for training but improve generalisation using a synthetic-to-real transformatoin GAN. However, all of this work requires supervision by ground truth depth. An alternative branch of methods explore using self-supervision from augmented data. For example, binocular stereo pairs can provide a supervisory signal through consistency of cross projected images \cite{kendall2017end, garg2016unsupervised, godard2017unsupervised}. Alternatively, video data can provide a similar source of supervision \cite{zhou2017unsupervised, vijayanarasimhan2017sfm, wang2018learning,monodepth2}. Tulsiani \etal \cite{tulsiani2017multi} use multiview supervision in a ray tracing network. While all these methods take single image input, Ji \etal \cite{ji2017surfacenet} tackle the MVS problem itself using deep learning.  

\mypara{Deep intrinsic image decomposition}
Intrinsic image decomposition is a partial step towards inverse rendering. It decomposes an image into reflectance (albedo) and shading but does not separate shading into shape and illumination. Even so, the lack of ground truth training data makes this a hard problem to solve with deep learning. Recent work either uses synthetic training data and supervised learning \cite{narihira2015direct,han2018learning, lettry2018darn, e.20181172,fan2017revisiting} or self-supervision/unsupervised learning. Very recently, Li \etal \cite{BigTimeLi18} used uncontrolled time-lapse images allowing them to combine an image reconstruction loss with reflectance consistency between frames. This work was further extended using photorealistic, synthetic training data \cite{li2018cgintrinsics}. Ma \etal \cite{ma2018single} also trained on time-lapse sequences and introduced a new gradient constraint which encourage better explanations for sharp changes caused by shading or reflectance. Baslamisli \etal \cite{Baslamisli_2018_CVPR} applied a similar gradient constraint while they used supervised training. Shelhamer \etal \cite{shelhamer2015scene} propose a hybrid approach where a CNN estimates a depth map which is used to constrain a classical optimisation-based intrinsic image estimation.

\mypara{Deep inverse rendering}
To date, solving the full inverse rendering problem using deep learning has received relatively little attention. One line of work simplifies the problem by restricting to a single object class, e.g.~faces \cite{tewari2017mofa}, meaning that a statistical face model can constrain the geometry and reflectance estimates. This enables entirely self-supervised training. Shu \etal \cite{shu2017neural} extend this idea with an adversarial loss. Sengupta \etal \cite{sengupta2017sfsnet} on the other hand, initialise with supervised training on synthetic data, and fine-tuned their network in an unsupervised fashion on real images. Nestmeyer \etal \cite{nestmeyer2019structural} target realistic face relighting, combining an inverse rendering network with a renderer that combines both physics-based and learnt elements. Going beyond faces, Kanamori and Endo \cite{kanamori2018relighting} consider whole body inverse rendering for relighting. Here, occlusion of the illumination environment becomes important so they show how to infer a light transport map.
Aittala \etal \cite{aittala2016reflectance} restrict geometry to almost planar objects and lighting to a flash in the viewing direction under which assumptions they can obtain impressive results. Gao \etal \cite{Gao:2019:DIR} consider the same planar scenario but with multiple images, enabling recovery of spatially varying reflectance properties.
More general settings have been considered including natural illumination \cite{li2017modeling}. Philip \etal \cite{PGZED19} focus on relighting of outdoor scenes but require multiple images and a 3D geometric proxy.
Kulkarni \etal \cite{kulkarni2015deep} show how to learn latent variables that correspond to extrinsic parameters allowing image manipulation. 
The only prior work we are aware of that tackles the full inverse rendering problem requires direct supervision \cite{janner2017self,liu2017material,li2018learning}. Hence, it is not applicable to scene-level inverse rendering, only objects, and relies on synthetic data for training, limiting the ability of the network to generalise to real images. Very recently, Sengupta \etal \cite{sengupta2019neural} combined an inverse rendering network with a trainable, residual rendering network. By pre-training both on synthetic data in a supervised fashion, they can subsequently finetune the inverse rendering network on real data using self-supervision (with the residual rendering network kept fixed). Li \etal \cite{li2020inverse} trained a cascade of inverse rendering networks in which reflectance is modelled as a spatially-varying BRDF and global lighting as a spatially-varying spherical Gaussian. The need for synthetic training data limited these approaches to indoor scenes.

\begin{figure*}[t]
    \centering
    \includegraphics[width=\textwidth,clip=true,trim=15px 250px 30px 280px]{diagram/inference.pdf}
    \caption{At inference time, our network regresses shadow, diffuse albedo and normal maps from a single, uncontrolled image. These are used to infer shadow free and shading only images from which we compute least squares optimal spherical harmonic lighting coefficients. At training time, we introduce self-supervision via an appearance loss computed using a differentiable renderer and the estimated quantities.}
    \label{fig:inference}
\end{figure*}

\section{Preliminaries}

In this section we define notation, introduce basic models and assumptions and describe our shape representation.

We assume that a perspective camera observes a scene, such that the projection from 3D world coordinates, $(u,v,w)$, to 2D image coordinates, $(x,y)$, is given by:
\small
\begin{equation}
    \lambda \begin{bmatrix}x\\y\\1\end{bmatrix} = 
    \mathbf{P}
    \begin{bmatrix}u\\v\\w\\1\end{bmatrix},\quad \mathbf{P} = \mathbf{K}
    \begin{bmatrix}\mathbf{R} & \mathbf{t}\end{bmatrix},\quad \mathbf{K} = \begin{bmatrix}f & 0 & c_x \\ 0 & f & c_y \\ 0 & 0 & 1\end{bmatrix},
\end{equation}
\normalsize
where $\lambda$ is an arbitrary scale factor, $\mathbf{R}\in SO(3)$ a rotation matrix, $\mathbf{t}\in\R^3$ a translation vector, $f$ the focal length and $(c_x,c_y)$ the principal point. 

The inverse rendered shape estimate could be represented in a number of ways. For example, many previous methods estimate a viewer-centred depth map. However, local reflectance, and hence appearance, is determined by surface orientation, i.e.~the local surface normal direction. So, to render a depth map for self-supervision, we would need to compute the surface normal. From a perspective depth map $w(x,y)$, the surface normal direction in camera coordinates is given by:
\begin{equation}
    \bar{\mathbf{n}} = \begin{bmatrix}-fw_x(x,y)\\-fw_y(x,y)\\(x-c_x)w_x(x,y) + (y-c_y)w_y(x,y) + w(x,y)\end{bmatrix}\label{eqn:perspnorm}
\end{equation}
from which the unit length normal is given by:
$    \mathbf{n}=\bar{\mathbf{n}}/\|\bar{\mathbf{n}}\|
$. The derivatives of the depth map in the image plane, $w_x(x,y)$ and $w_y(x,y)$, can be approximated by finite differences. However, \eqref{eqn:perspnorm} requires knowledge of the intrinsic camera parameters. This would severely restrict the applicability of our method. For this reason, we choose to estimate a surface normal map directly. An additional advantage of working in the surface normal domain is that the normal map can capture high frequency shape details without strictly satisfying integrability constraints. This leads to higher quality image renderings.

Although the surface normal can be represented by a 3D vector, since $\|\mathbf{n}\|_2=1$ it has only two degrees of freedom. So, our network estimates two quantities per-pixel: $n_x/n_z$ and $n_y/n_z$. Since $n_z>0$ for visible pixels we can compute the surface normal from the estimated quantities as:
\begin{equation}
    \mathbf{n} = \frac{[n_x/n_z,n_y/n_z,1]^T}{\|[n_x/n_z,n_y/n_z,1]\|}.\label{eqn:gradnorm}
\end{equation}

We assume that appearance can be approximated by a local reflectance model under environment illumination, modulated by a scalar shadowing/ambient occlusion term. Specifically we use a Lambertian diffuse model with order 2 spherical harmonic lighting. 

Alone, this cannot model phenomena such as cast shadows, spatially varying illumination and specularities. In \cite{yu2019inverserendernet}, this results in these phenomena being baked into one or both of the albedo and normal maps. Of these phenomena, the most severe are cast shadows. We introduce an additional term that acts multiplicatively on the appearance predicted by the local spherical harmonics model. Without appropriate constraint, the introduction of this additional channel could lead to trivial solutions. Hence, we constrain it in two ways. First, we restrict it to the range $[0,1]$ so that it can only downscale appearance. Second, it is a scalar quantity acting equally on all colour channels. Together, these restrictions encourage this channel to explain cast shadows and we refer to it as a shadow map. However, note that we do not expect it to be a physically valid shadow map nor that it contains only shadows.

Under this model, RGB intensity can be computed as
\begin{equation}
    \mathbf{i}(x,y) = \begin{bmatrix}i_r(x,y)\\i_g(x,y)\\i_b(x,y) \end{bmatrix} = \bm{\alpha}(x,y)\odot s(x,y)\mathbf{B}(\mathbf{n}(x,y))\mathbf{l},\label{eqn:imageform}
\end{equation}
where $\odot$ is the Hadamard (element-wise) product, $\mathbf{l}\in\R^{27}$ contains the order 2, colour spherical harmonic colour illumination coefficients, $\bm{\alpha}(x,y)=[\alpha_r(x,y),\alpha_g(x,y),\alpha_b(x,y)]^T$ is the colour diffuse albedo, $s(x,y)\in [0,1]$ is the shadowing weight and the order 2 basis $\mathbf{B}(\mathbf{n})\in\R^{3\times 27}$ is given by $\mathbf{B}(\mathbf{n})=\mathbf{I}_{3}\otimes\mathbf{b}(\mathbf{n})$ where $\otimes$ is the Kronecker product and
{\small \begin{equation}
    \mathbf{b}(\mathbf{n})=[1,n_x,n_y,n_z,3n_z^2-1,n_xn_y,n_xn_z,n_yn_z,n_x^2-n_y^2].\label{eqn:basis}
\end{equation}}
This appearance model neglects high frequency illumination effects and interreflections. However, we found that in practice this model works well for typical outdoor scenes.

\section{Architecture}

Our inverse rendering network (see Fig.~\ref{fig:inference}) is an image-to-image network that regresses albedo, surface normal and shadow maps from a single image and uses these to estimate lighting. We now describe these components in more detail.

\subsection{Trainable encoder-decoder}

We implement a deep fully-convolutional neural network with skip connections based on the U-net architecture \cite{ronneberger2015u}. The specific choice of architecture is orthogonal to our contribution and we have experimented with using different variations without significant effect on the result. We use a single encoder and separate deconvolution decoders for albedo, shadow and normal map prediction. Albedo maps have 3 channel RGB output, normal maps have two channels that are converted to a normal map using \eqref{eqn:gradnorm} and shadow maps have a single channel. Both albedo and shadow maps are transformed to the $[0,1]$ range by applying the hyperbolic tangent function followed by a scaling and offset. Both the convolutional subnetwork and deconvolutional subnetwork contain 15 layers and the activation functions are ReLUs.

\subsection{Illumination separability}

In order to estimate illumination parameters, one option would be to use a fully connected branch from the output of our encoder and train our network to predict it directly. Sengupta \etal \cite{sengupta2017sfsnet} use such an approach by training a small CNN that takes image, albedo and normal features as input and uses $1\times 1$ convolutions, average pooling and a fully connected layer to predict lighting. While such a network can potentially learn robust lighting estimation, it is a black box whose behaviour is not interpretable and there is no guarantee that the estimated illumination is optimal with respect to the estimated image, albedo and normals. We tried training such a network as part of our inverse rendering network but training did not converge - perhaps because of our more challenging unconstrained outdoor scenes as opposed to the face images used in \cite{sengupta2017sfsnet}. We observe that lighting can be inferred from the input image and estimated albedo, shadow and normal maps, making its explicit prediction redundant. We avoid this redundancy by computing least squares optimal lighting in-network from the estimated maps. An additional advantage of this approach is that the architecture remains fully convolutional and so can process images of any size at inference time.

We detect and mask the sky region by applying PSPNet \cite{zhao2017pyramid}. The sky region does not adhere to our reflectance model and albedo and normal maps are meaningless in these regions. Hence, we use only the non-sky foreground in illumination estimation. 

For an input image with $K$ foreground pixels, we stack the $K$ RGB values to form the vector $\mathbf{i}_{\text{obs}}\in\R^{3K}$. We assume that nonlinear gamma has been applied with a fixed $\gamma=2.2$. We invert the nonlinear gamma and equate with our model from \eqref{eqn:imageform} extended to the whole image:
\begin{align}
    \mathbf{i}_{\text{obs}}^{\gamma} &= [i_r^1,\dots,i_r^K,i_g^1,\dots,i_g^K,i_b^1,\dots,i_b^K]^T \notag \\
    &=\bm{\alpha} \odot (\mathbf{1}_{3\times 1} \otimes \mathbf{s}) \odot \mathbf{B}\mathbf{l} \label{eqn:illummodim}
\end{align}
where each row of $\mathbf{B}\in\R^{3K\times 27}$ contains the spherical harmonic basis for the estimated surface normal at one pixel and the foreground pixels from the estimated albedo and shadow maps are stacked to form $\bm{\alpha}\in\R^{3K}$ and $\mathbf{s}\in\R^K$ respectively. We can now rearrange \eqref{eqn:illummodim} into linear least squares form with respect to the lighting vector:
\begin{equation}
    \mathbf{i}_{\text{obs}}^{\gamma} = \left[ \mathbf{1} \otimes (\bm{\alpha} \odot (\mathbf{1}_{3\times 1} \otimes \mathbf{s})) \odot \mathbf{B} \right] \mathbf{l} = \mathbf{Al}.\label{eqn:lsqlight}
\end{equation}
We can now solve for the spherical harmonic illumination coefficients in a least squares sense over all foreground pixels. This can be done using any method, so long as the computation is differentiable such that losses dependent on the estimated illumination can have their gradients backpropagated into the inverse rendering network. We solve using the pseudoinverse of $\mathbf{A}$, i.e.~$\mathbf{l}^*=\mathbf{A}^+\mathbf{i}_{\text{obs}}^{\gamma}$. Note that the pseudoinverse $\mathbf{A}^+$ has a closed form derivative \cite{golub1973differentiation}. Fig.~\ref{fig:inference} shows the inferred normal, albedo and shadow maps, and a visualisation of the least squares estimated lighting.

\section{Data and preprocessing}\label{sec:data}

\begin{figure}[!t]
    \centering
\begingroup
\setlength{\tabcolsep}{1pt}
\renewcommand{\arraystretch}{0.5}
\small{
\resizebox{\linewidth}{!}{
\begin{tabular}{cccc}
Input & \footnotesize{Ground Plane} & MVS Normal & New Normal\\
\includegraphics[width=2.cm]{ground_plane/3/im.png}& 
\includegraphics[width=2.cm]{ground_plane/3/mask_gp.png}&
\includegraphics[width=2.cm]{ground_plane/3/nm_old.png}&
\includegraphics[width=2.cm]{ground_plane/3/nm.png}
\\
\end{tabular}
}}
\endgroup
    \caption{The mask for ground plane is obtained by PSPNet \cite{zhao2017pyramid}. We then define the normal for masked ground plane by the normal of fitted camera plane.}
    \label{fig:ground_plane}
\end{figure}

In order to supervise the training of our network, we begin by applying a multiview reconstruction pipeline to large, uncontrolled, outdoor image collections. The application of structure-from-motion followed by multiview stereo (which we refer to simply as MVS) enables both camera poses and dense 3D scene models to be estimated from large, uncontrolled image sets. Of particular importance to us, these pipelines are relatively insensitive to illumination variation between images in the dataset since they rely on matching local image features that are themselves illumination insensitive. 
We use a modification of the recent COLMAP \cite{schonberger2016structure} pipeline. 
We emphasise that MVS is run offline prior to training and that at inference time our network uses only single images of novel scenes. We use the MVS output for three sources of supervision.

From the MVS geometry, we can compute per-view depth maps and subsequently estimate surface normal maps using \eqref{eqn:perspnorm}. These normal maps are used as coarse direct supervision during training. However, the MVS models almost always do not reconstruct the ground plane. We find that this significantly disrupts training such that our network does not learn to predict good shape or albedo estimates in the ground plane region. For this reason, we preprocess the MVS normal maps to replace ground plane normals with an assumed ground plane normal direction. To this end, we detect ground plane pixels using PSPNet \cite{zhao2017pyramid} and inpaint normal direction to detected pixels in the normal map. Although the true ground plane direction is unknown, a reasonable estimate can be made from the MVS reconstruction. We assume that the cameras positions are located approximately a fixed height above the ground and fit a plane to their positions using principal components analysis (PCA). The normal to this plane defines the ground plane normal which we rotate into camera coordinates and use for inpainting. Note, the estimate of the ground plane orientation is done using the whole MVS reconstruction for a scene and is the same for all images in that scene. Hence, if a particular image has large parts of the ground plane occluded, the estimated ground plane for that image is not disrupted. The only per-image part of the process is estimating the ground segmentation that defines which pixels have their normal directions replaced by the estimated ground plane orientation. A sample result is shown in Fig.~\ref{fig:ground_plane}. 

Our primary dataset is from MegaDepth \cite{MegaDepthLi18}. This contains dense depth maps and camera calibration parameters estimated from crawled Flickr images. The pre-processed images have arbitrary shapes and orientations. For ease of training, we resize images such that the smaller side is of size $200$ pixels, then crop multiple $200\times 200$ square images. We choose our crops to maximise the number of pixels with defined depth values. Where possible, we crop multiple images from each image, achieving augmentation as well as standardisation. We adjust the camera parameters to account for the scale and crop. We create mini-batches in which all pairs of images within the mini-batch have overlapping views (defined as having similar camera locations and similar centres of mass of their backprojected depth maps). We also enforce sufficient illumination variation by cross-projecting (see Section \ref{sec:crossproj}) each image into the others and discarding pairs where the correlation coefficient of intensity histograms is close to 1). Finally, before inputting an image to our network, we detect and mask the sky region using PSPNet \cite{zhao2017pyramid}. Apart from this, we input the images as stored, without undoing the assumed nonlinear gamma. In total, we use 117,030 images from MegaDepth. For parts of our evaluation, we finetune using additional datasets (described in Section \ref{sec:eval}). These are preprocessed in the same way.

\section{Losses and training strategy}

As shown in Fig.~\ref{fig:inference}, we use a data term (the error between predicted and observed appearance) for self-supervision. However, inverse rendering using only a data term is ill-posed (an infinite set of solutions can yield zero data error) and so we use additional sources of supervision, all of which are essential for good performance. We describe all sources of supervision in this section.

\subsection{Self-supervision via differentiable rendering}

Given estimated normal, shadow and albedo maps and spherical harmonic illumination coefficients, we compute a predicted image using \eqref{eqn:imageform}. This local illumination model is straightforward to differentiate. Self-supervision is provided by the error between the predicted and observed RGB colour values. We found that performance was significantly improved by computing this error in a shadow-free space. To do so, we divide out the estimated shadow map from the linearised observed image and clamp to one (avoiding numerical instabilities caused by very small shadow values):
\begin{equation}
    \mathbf{i}_{\text{SF}} = \min\left[1,\mathbf{i}_{\text{obs}}^{\gamma}\oslash \left(\mathbf{1}_{3\times 1} \otimes \mathbf{s}\right) \right],
\end{equation}
where $\oslash$ is Hadamard (element-wise) division. We compare this shadow free image to the image predicted using only the local spherical harmonic model and our estimated illumination and albedo and normal maps. Inspired by \cite{zhang2016colorful}, computing this appearance loss directly in RGB space does not yield the best results since it is not the best measure of the perceptual quality of the reconstruction. For this reason, we use two transformations to perceptually meaningful feature spaces and compute an L2 loss in those spaces:
\begin{equation}
\ell_{\textrm{appearance}} = \varepsilon(\bm{\alpha} \odot \mathbf{B}\mathbf{l},\mathbf{i}_{\text{SF}}),\label{eqn:apploss}
\end{equation}
where
\begin{multline}
\varepsilon(\mathbf{x},\mathbf{y})=
    w_{\textrm{VGG}}\left\| \text{VGG}(\mathbf{x})-\text{VGG}(\mathbf{y}) \right\|_2 +\\
    w_{\textrm{LAB}}\left\| \text{LAB}(\mathbf{x})-\text{LAB}(\mathbf{y}) \right\|_2,
\end{multline}
where VGG computes features from the first two convolution blocks of a pre-trained VGG network \cite{johnson2016perceptual} and LAB transforms to the LAB colour space. We mask these losses pixel-wise using the sky mask, using appropriate downsampling of the mask within the VGG layers. We set the weights as $w_{\textrm{VGG}}=2.5$ and $w_{\textrm{LAB}}=0.5$ in all experiments.

\begin{figure}
    \centering
        \begin{minipage}[c]{.33\linewidth}
            \begin{subfigure}[b]{\linewidth}
                \centering
                \includegraphics[width=0.9\linewidth]{illu_model/+pc1.jpg}
                \caption{mean\,+\,1st}
            \end{subfigure}\vspace{10pt}
            
            \begin{subfigure}[b]{\linewidth}
                \centering
                \includegraphics[width=0.9\linewidth]{illu_model/+pc2.jpg}
                \caption{mean\,+\,2nd}
            \end{subfigure}
        \end{minipage}% %first
        \begin{minipage}[c]{.33\linewidth}
            \begin{subfigure}[b]{\linewidth}
                \centering
                \includegraphics[width=0.9\linewidth]{illu_model/_pc3.jpg}
                \caption{mean\,-\,3rd}
            \end{subfigure}\vspace{-0pt}
            
            \begin{subfigure}[b]{\linewidth}
                \centering
                \includegraphics[width=0.9\linewidth]{illu_model/mean.jpg}
                \caption{mean}
            \end{subfigure}\vspace{-0pt}
            
            \begin{subfigure}[b]{\linewidth}
                \centering
                \includegraphics[width=0.9\linewidth]{illu_model/+pc3.jpg}
                \caption{mean\,+\,3rd}
            \end{subfigure}
        \end{minipage}% % second
        \begin{minipage}[c]{.33\linewidth}
            \begin{subfigure}[b]{\linewidth}
                \centering
                \includegraphics[width=0.9\linewidth]{illu_model/_pc2.jpg}
                \caption{mean\,-\,2nd}
            \end{subfigure}\vspace{10pt}
            
            \begin{subfigure}[b]{\linewidth}
                \centering
                \includegraphics[width=0.9\linewidth]{illu_model/_pc1.jpg}
                \caption{mean\,-\,1st}
            \end{subfigure}
        \end{minipage} % %third
        \caption{Statistical illumination model. The central image shows the mean illumination. The two diagonals and the vertical show the first 3 principal components.} 
 \label{fig:illu_model}     
 \end{figure}
 
\subsection{Natural illumination model and prior}

The spherical harmonic lighting model in \eqref{eqn:imageform} enables efficient representation of complex lighting. However, even within this low dimensional space, not all possible illumination environments are natural. The space of natural illumination possesses statistical regularities \cite{dror2001statistics}. We can use this knowledge to constrain the space of possible illumination and enforce a prior on the illumination parameters. To do this, we build a statistical illumination model (see Fig.~\ref{fig:illu_model}) using a dataset of 79 HDR spherical panoramic images taken outdoors \cite{hdrlabs,hdriskies}. For each environment, we compute the spherical harmonic coefficients, $\mathbf{l}_i\in\R^{27}$. Since the overall intensity scale is arbitrary, we also normalise each lighting matrix to unit norm, $\|\mathbf{l}_i\|_2=1$, to avoid ambiguity with the albedo scale. Our illumination model in \eqref{eqn:illummodim} uses surface normals in a viewer-centred coordinate system. So, the dataset must be augmented to account for possible rotations of the environment relative to the viewer. Since the rotation around the vertical ($v$) axis is arbitrary, we rotate the lighting coefficients by angles between $0$ and $2\pi$ in increments of $\pi/18$. In addition, to account for camera pitch or roll, we additionally augment with rotations about the $u$ and $w$ axes in the range $(-\pi/6,\, \pi/6)$. This gives us a dataset of 139,356 environment maps.
We then build a statistical model, such that any illumination can be approximated as:
\begin{equation}
    \mathbf{l} = \mathbf{Q}\textrm{diag}(\sigma_1,\dots,\sigma_D){\bm \alpha} + \mathbf{\bar{l}}.\label{eqn:illummod}
\end{equation}
where $\mathbf{Q}\in\R^{27\times D}$ contains the principal components, $\sigma^2_1,\dots,\sigma^2_D$ are the corresponding eigenvalues, $\mathbf{\bar{l}}\in\R^{27}$ is the mean lighting coefficients and ${\bm \alpha}\in\R^D$ is the parametric representation of $\mathbf{l}$. We use $D=18$ dimensions. Under the assumption that the original data is Gaussian distributed then the parameters are normally distributed: ${\bm \alpha}\sim\cal{N}(\mathbf{0},\mathbf{I})$. When we compute lighting, we do so within the subspace of the statistical model. i.e.~we substitute \eqref{eqn:illummod} into \eqref{eqn:lsqlight} and solve linearly for ${\bm \alpha}$. In addition, we introduce a prior loss on the estimated lighting vector: $\ell_{\textrm{lighting}} = \| {\bm \alpha} \|_2^2$.

\subsection{Multiview stereo supervision}


\begin{figure*}[t]
    \centering
    \includegraphics[width=\linewidth,clip=true,trim=23px 217px 58px 325px]{diagram/cross_rendering_3.pdf}
    \caption{Cross-rendering loss. A cross-rendered image is generated by relighting the albedo and normal predictions from one image and the lighting prediction from the other. Before rendering, we rotate the lighting to align it with the new view given relative camera poses. The rendering is shadow free, so the shadow is removed from ground truth relighting image. Both shadow and ground truth relighting image are cross-projected from the view from which new lighting is taken.}
    \label{fig:cross_rendering}
\end{figure*}

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth,clip=true,trim=100px 310px 120px 230px]{diagram/cross_proj_2.pdf}
    \caption{Albedo consistency loss. Given the depth map and camera parameters from performing MVS over image collections, the albedo prediction from one view can be cross-projected to another view. For each image, we compute the albedo consistency loss by measuring the difference between albedo prediction and cross-projected albedo prediction from other views within each batch.}
    \label{fig:cross_proj}
\end{figure}


\mypara{Cross-projection}\label{sec:crossproj}
We use the MVS poses and depth maps to establish correspondence between views, allowing us to cross-project quantities between overlapping images. Given an estimated depth map, $w(x,y)$, in view $i$ and camera matrices for views $i$ and $j$, a pixel $(x,y)$ can be cross-projected to location $(x^{\prime},y^{\prime})$ in view $j$ via:
\begin{equation}
\lambda\begin{bmatrix}
x^{\prime}\\
y^{\prime}\\
1
\end{bmatrix}=
    %\mathbf{P}_j
    \mathbf{K}_j
    \begin{bmatrix}
    \mathbf{R}_j & \mathbf{t}_j
    \end{bmatrix}
    \begin{bmatrix}\mathbf{R}_i^T&-\mathbf{R}_i^T\mathbf{t}_i\\ \mathbf{0} & 1\end{bmatrix}
    \begin{bmatrix}
    w(x,y)\mathbf{K}_i^{-1}\begin{bmatrix}x\\y\\1\end{bmatrix}\\
    1\end{bmatrix}
    %\begin{bmatrix}\frac{x-c_x}{f}w(x,y)\\\frac{y-c_y}{f}w(x,y)\\w(x,y)\\1\end{bmatrix},
\end{equation}
In practice, we perform the cross-projection in the reverse direction, computing non-integer pixel locations in the source view for each pixel in the target view. We can then use bilinear interpolation of the source image to compute quantities for each pixel in the target image. Since the MVS depth maps contain holes, any pixels that cross project to a missing pixel are not assigned a value. Similarly, any target pixels that project outside the image bounds of the source are not assigned a value. We denote the cross-projection of any quantity $\mathbf{x}_i$ defined on the pixels of image $i$ onto the pixels of image $j$ as $\text{proj}_{i\rightarrow j}(\mathbf{x}_i)$.


\mypara{Direct normal map supervision}
The per-view depth maps provided by MVS can be used to estimate normal maps, albeit that they are typically coarse and incomplete (see e.g.~Fig.~\ref{fig:ground_plane}, col.~4). We compute guide normal maps from the depth maps and intrinsic camera parameters estimated by MVS using \eqref{eqn:perspnorm}. The guide normal maps, $\mathbf{n}_{\textrm{guide}}$, are used for direct supervision by computing an angular error loss $\ell_{\textrm{NM}} = \arccos(\mathbf{n}_{\textrm{guide}}\cdot \mathbf{n}_{\textrm{est}})$ where $\mathbf{n}_{\textrm{est}}$ are the estimated surface normals. This loss is summed over foreground pixels in the supervision depth map.

\mypara{Albedo consistency loss}
Diffuse albedo is an intrinsic quantity. Hence, we expect that albedo estimates of the same scene point from two overlapping images should be the same, even if the illumination varies between views. We train our network in a Siamese fashion on overlapping pairs within each mini-batch and use cross projection to compute an albedo consistency loss, again computing the loss as the sum of two perceptual losses:
\begin{equation}
    \ell_{\textrm{albedo}} = \varepsilon({\bm \alpha}_i,\text{proj}_{j\rightarrow i}({\bm \alpha}_j)),
\end{equation}
where ${\bm \alpha}_i$ and ${\bm \alpha}_j$ are the estimated albedo maps in the $i$th and $j$th images respectively. This loss is masked both by the sky mask in image $i$ and by the cross-projected pixels for which image $j$ has a defined MVS depth value. The weights $w_{\textrm{VGG}}$ and $w_{\textrm{LAB}}$ are set the same as in \eqref{eqn:apploss}. The albedo consistency loss is visualised by the blue arrows in Fig.~\ref{fig:cross_proj}.

\mypara{Cross-rendering loss} We also use a hybrid of cross-projection and appearance loss. This remedies a deficiency in using albedo consistency alone. Cross-projection at low (network input) resolution yields blurred output. The result is that the albedo consistency loss encourages blurred (but consistent) albedo maps. We formulate another loss that encourages albedo consistency but which does not require cross projection of low resolution albedo maps. This cross-rendering loss also helps disentangle scenes into view-dependent illumination and physically-constant diffuse albedo. Here, we recombine predicted albedo with illumination estimated from another view, whose appearance should be reproduced by such a recombination.

We cross-project the images in the dataset at their original high resolution, using the high resolution depth maps. We define the cross projection of image $j$ into image $i$ using the original high resolution image, $\mathbf{i}_{j}^{\text{HR}}$, in view $j$ as:
\begin{equation}
    \mathbf{i}_{j\rightarrow i}=\text{downsample}(\text{proj}_{j\rightarrow i}(\mathbf{i}_{j}^{\text{HR}})),
\end{equation}
where the downsample function downsamples to network input resolution. Note that this cross projection need only be done once as preprocessing step on the training dataset.

We then mix the albedo and normal predictions from image $i$ with the lighting and cross projected shadow map from image $j$ and compare it against the high resolution cross projection from image $j$. Again, we do so in a shadow free space, so define the cross-projected shadow free image:
\begin{equation}
    \mathbf{i}_{j\rightarrow i}^{\text{SF}} = \min\left[1,\mathbf{i}_{j\rightarrow i} \oslash \left(\mathbf{1}_{3\times 1} \otimes \text{proj}_{j\rightarrow i}(\mathbf{s}_j)\right) \right],
\end{equation}
and then compute the cross-rendering loss as:
\begin{equation}
    \ell_{\textrm{cross-rend}} = \varepsilon(\mathbf{i}_{j\rightarrow i}^{\text{SF}},\bm{\alpha}_i \odot \mathbf{B}_i\mathbf{R}_{j\rightarrow i}\mathbf{l}_j),
\end{equation}
where $\mathbf{B}_i$ is the spherical harmonic basis computed from the normal map estimated from image $i$, $\mathbf{l}_j$ is the spherical harmonic lighting coefficients estimated in image $j$ and $\mathbf{R}_{j\rightarrow i}$ rotates the coefficients from the camera coordinate system in image $j$ to image $i$. 
We visualise the process of computing this loss in Fig.~\ref{fig:cross_rendering}. 


\subsection{Training}

We train our network to minimise:
\begin{multline}
    \ell = w_1\ell_{\textrm{appearance}} + w_2\ell_{\textrm{NM}} + w_3\ell_{\textrm{albedo}} + \\w_4\ell_{\textrm{cross-rend}} +  w_5\ell_{\textrm{lighting}}
\end{multline}
where the weights are set as $w_1=0.1$, $w_2=1.0$, $w_3=0.1$, $w_4=0.1$ and $w_5=0.005$.

We use the Adam optimiser \cite{kingma2014adam}. We found that for convergence to a good solution it is important to include a pre-training phase. During this phase, the surface normals used for illumination estimation and for the appearance-based losses are the MVS normal maps. This means that the surface normal prediction decoder is only learning from the direct supervision loss, i.e.~it is learning to replicate the MVS normals. After this initial phase, we switch to full self-supervision where the predicted appearance is computed entirely from estimated quantities.

\begin{table}[!t]
\centering
\setlength{\tabcolsep}{3pt}
\renewcommand{\arraystretch}{1}
    \begin{tabular}{c|cc|cc}
    \hline
    \multirow{2}{*}{Method} & \multicolumn{2}{c|}{Reflectances} & \multicolumn{2}{c} {Normals} \\ 
      & MSE & LMSE & Mean & Median \\
    \hline
    \hline
    MegaDepth \cite{MegaDepthLi18}  & - & -  & 43.6 & 43.0 \\ % new nm
    Godard \etal \cite{godard2017unsupervised}  & - & - & 82.4 & 82.1 \\ % new nm
    Nestmeyer \etal \cite{nestmeyer2017reflectance}  & 0.0149 & 0.0169 & - & - \\
    BigTime \cite{BigTimeLi18}  & 0.0116 & 0.0135 & - & - \\
    SIRFS \cite{BarronTPAMI2015} & \textbf{0.0070} & 0.0275 & 50.6 & 48.5 \\
    InverseRenderNet \cite{yu2019inverserendernet}  & 0.0112 & 0.0128 & 40.0 & 38.0 \\ % new nm
    ours  & 0.0093 & \textbf{0.0111} & \textbf{31.2} & \textbf{30.0} \\ % 22-38
    \hline
    \end{tabular}
\caption{Quantitative inverse rendering results on MegaDepth dataset \cite{MegaDepthLi18}. Reflectance (albedo) errors are measured against multiview inverse rendering result \cite{kim2016multi} with optimal per-channel scaling applied and normals against MVS results.}
\label{tab:am&nm_table}
\end{table}

\begin{table}[!t]
\centering
\setlength{\tabcolsep}{2pt}
\renewcommand{\arraystretch}{1}
    \begin{tabular}{c|cc|cc}
    \hline
    \multirow{2}{*}{Method} & \multicolumn{2}{c|}{Reconstruction} & \multicolumn{2}{c}{Consistency}\\
     & MD \cite{MegaDepthLi18} & BT \cite{BigTimeLi18} & MD \cite{MegaDepthLi18} & BT \cite{BigTimeLi18} \\
    \hline
    \hline
    Nestmeyer \etal \cite{nestmeyer2017reflectance} & 0.1003 & 0.0333 & 0.0140 & \textbf{0.0067}\\
    BigTime \cite{BigTimeLi18}  & 0.0372 & - & 0.0089 & - \\
    InverseRenderNet \cite{yu2019inverserendernet}  & 0.0158 & 0.0124 & 0.0168 & 0.0118\\
    ours  & \textbf{0.0082} & \textbf{0.0082} & \textbf{0.0082} & 0.0072 \\ % 22-38
    \hline
    \end{tabular}
\caption{Reconstruction error and reflectance consistency for MegaDepth \cite{MegaDepthLi18} and BigTime dataset \cite{BigTimeLi18}. MSE is used for computing reconstruction errors, and MSE with optimal scaling is used for computing albedo consistency.}
\label{tab:recon&consis}
\end{table}

\section{Evaluation}\label{sec:eval}

There are no existing benchmarks for inverse rendering of outdoor scenes, partly because there is no existing method for satisfactorily estimating scene-level geometry, reflectance and lighting for in-the-wild, outdoor scenes. For this reason, we developed our own outdoor benchmark based on the MegaDepth dataset \cite{MegaDepthLi18}. We augment this with four further benchmarks that evaluate performance of a subset of the network outputs. First, we use the BigTime timelapse dataset \cite{BigTimeLi18} to evaluate albedo consistency under varying illumination. Second, we evaluate on the related task of intrinsic image decomposition using an indoor benchmark \cite{bell14intrinsic} which demonstrates the generalisation ability of our network. Third, we quantify our normal prediction using the DIODE benchmark \cite{diode_dataset}, which contains high quality outdoor depth and normal maps along with corresponding images. Finally, we evaluate illumination estimation on the benchmark dataset in \cite{yu20relightNet} consisting of multi-illumination and multi-view images along with ground truth HDR illumination environment maps.


\begin{figure}[!t]
    \centering
\begingroup
\setlength{\tabcolsep}{1pt}
\renewcommand{\arraystretch}{0.5}
\resizebox{\linewidth}{!}{
\begin{tabular}{ccccc}
\multirow{2}{*}{Input} & {\scriptsize BigTime}  & {\scriptsize Nestmeyer \etal}  & {\scriptsize InverseRenderNet} & \multirow{2}{*}{Ours}\\
& {\scriptsize \cite{BigTimeLi18}} & {\scriptsize \cite{nestmeyer2017reflectance}} & {\scriptsize \cite{yu2019inverserendernet}} & \\

\includegraphics[width=2.cm]{recon/2/im.png}& 
\includegraphics[width=2.cm]{recon/2/am_ww.png}&
\includegraphics[width=2.cm]{recon/2/am_filter.png}&
\includegraphics[width=2.cm]{recon/2/am_irn.png}&
\includegraphics[width=2.cm]{recon/2/am_ours.png}
\\
& 
\includegraphics[width=2.cm]{recon/2/shading_ww.png}&
\includegraphics[width=2.cm]{recon/2/shading_filter.png}&
\includegraphics[width=2.cm]{recon/2/shading_irn.png}&
\includegraphics[width=2.cm]{recon/2/shading_ours.png}
\\
&
\includegraphics[width=2.cm]{recon/2/recon_ww.png}&
\includegraphics[width=2.cm]{recon/2/recon_filter.png}&
\includegraphics[width=2.cm]{recon/2/recon_irn.png}&
\includegraphics[width=2.cm]{recon/2/recon_ours.png}
\\
\\
\includegraphics[width=2.cm]{recon/3/im.png}& 
\includegraphics[width=2.cm]{recon/3/am_ww.png}&
\includegraphics[width=2.cm]{recon/3/am_filter.png}&
\includegraphics[width=2.cm]{recon/3/am_irn.png}&
\includegraphics[width=2.cm]{recon/3/am_ours.png}
\\
& 
\includegraphics[width=2.cm]{recon/3/shading_ww.png}&
\includegraphics[width=2.cm]{recon/3/shading_filter.png}&
\includegraphics[width=2.cm]{recon/3/shading_irn.png}&
\includegraphics[width=2.cm]{recon/3/shading_ours.png}
\\
&
\includegraphics[width=2.cm]{recon/3/recon_ww.png}&
\includegraphics[width=2.cm]{recon/3/recon_filter.png}&
\includegraphics[width=2.cm]{recon/3/recon_irn.png}&
\includegraphics[width=2.cm]{recon/3/recon_ours.png}
\\

\end{tabular}
}
\endgroup
    \caption{Qualitative results for reconstruction. The first and fourth rows are albedo predictions, and second and fifth rows are shading predictions from labelled methods. The reconstruction results for each method is composed by albedo and shading and is shown on third and last rows.}
    \label{fig:recon}
\end{figure}



\begin{figure*}[!t]
    \centering
\begingroup
\setlength{\tabcolsep}{1pt}
\renewcommand{\arraystretch}{0.7}
\small{
\resizebox{\linewidth}{!}{
\begin{tabular}{ccccccccc}
 \multirow{2}{*}{\textbf{Input}}  & \multicolumn{4}{c}{\textbf{Albedo}} & \multicolumn{4}{c}{\textbf{Cross-projection}} \\
 & \cite{BigTimeLi18} & \cite{yu2019inverserendernet} & \cite{nestmeyer2017reflectance} & Ours & \cite{BigTimeLi18} & \cite{yu2019inverserendernet} & \cite{nestmeyer2017reflectance} & Ours \\
\includegraphics[width=2.cm]{am_consist/2/im1_sky.png}& 
\includegraphics[width=2.cm]{am_consist/2/am_ww1.png}&
\includegraphics[width=2.cm]{am_consist/2/am_irn1.png}&
\includegraphics[width=2.cm]{am_consist/2/am_filter1.png}&
\includegraphics[width=2.cm]{am_consist/2/am_ours1.png}&
\includegraphics[width=2.cm]{am_consist/2/am_ww1_cproj.png}&
\includegraphics[width=2.cm]{am_consist/2/am_irn1_cproj.png}&
\includegraphics[width=2.cm]{am_consist/2/am_filter1_cproj.png}&
\includegraphics[width=2.cm]{am_consist/2/am_ours1_cproj.png}
\\
\includegraphics[width=2.cm]{am_consist/2/im2_sky.png}& 
\includegraphics[width=2.cm]{am_consist/2/am_ww2.png}&
\includegraphics[width=2.cm]{am_consist/2/am_irn2.png}&
\includegraphics[width=2.cm]{am_consist/2/am_filter2.png}&
\includegraphics[width=2.cm]{am_consist/2/am_ours2.png}&
\includegraphics[width=2.cm]{am_consist/2/am_ww2_cproj.png}&
\includegraphics[width=2.cm]{am_consist/2/am_irn2_cproj.png}&
\includegraphics[width=2.cm]{am_consist/2/am_filter2_cproj.png}&
\includegraphics[width=2.cm]{am_consist/2/am_ours2_cproj.png}
\\
\vspace{-1mm}
\\
\includegraphics[width=2.cm]{am_consist/1/im1_sky.png}& 
\includegraphics[width=2.cm]{am_consist/1/am_ww1.png}&
\includegraphics[width=2.cm]{am_consist/1/am_irn1.png}&
\includegraphics[width=2.cm]{am_consist/1/am_filter1.png}&
\includegraphics[width=2.cm]{am_consist/1/am_ours1.png}&
\includegraphics[width=2.cm]{am_consist/1/am_ww1_cproj.png}&
\includegraphics[width=2.cm]{am_consist/1/am_irn1_cproj.png}&
\includegraphics[width=2.cm]{am_consist/1/am_filter1_cproj.png}&
\includegraphics[width=2.cm]{am_consist/1/am_ours1_cproj.png}
\\
\includegraphics[width=2.cm]{am_consist/1/im2_sky.png}& 
\includegraphics[width=2.cm]{am_consist/1/am_ww2.png}&
\includegraphics[width=2.cm]{am_consist/1/am_irn2.png}&
\includegraphics[width=2.cm]{am_consist/1/am_filter2.png}&
\includegraphics[width=2.cm]{am_consist/1/am_ours2.png}&
\includegraphics[width=2.cm]{am_consist/1/am_ww2_cproj.png}&
\includegraphics[width=2.cm]{am_consist/1/am_irn2_cproj.png}&
\includegraphics[width=2.cm]{am_consist/1/am_filter2_cproj.png}&
\includegraphics[width=2.cm]{am_consist/1/am_ours2_cproj.png}
\end{tabular}
}}
\endgroup
    \caption{Qualitative results for albedo consistency. For each consecutive pair of rows we show results for two overlapping images of the same scene. Albedo predictions are shown in col.~2-5 which are then cross-projected to the viewpoint of the other image in the pair in col.~6-9. Results shown for col.~2 and 6: BigTime \cite{BigTimeLi18}, col.~3 and 7: InverseRenderNet \cite{yu2019inverserendernet}, col.~4 and 8: Nestmeyer \etal \cite{nestmeyer2017reflectance}, col.~5 and 9: ours.}
    \label{fig:am_consist}
\end{figure*}

\subsection{Evaluation on MegaDepth}\label{sec:md_eval}

We split the MegaDepth dataset \cite{MegaDepthLi18} into training and testing data, and evaluate our performance on testing data. We evaluate the inverse rendering results through four aspects. Firstly, the normal estimation performance can be directly compared against the MVS geometry. We quantify performance using angular error in degrees. Second, we evaluate albedo estimation against the output a state-of-the-art multiview inverse rendering algorithm \cite{kim2016multi}. 
We are not aware of any existing method or device that can measure accurate scene-level albedo under in-the-wild conditions. Hence, we could not obtain ground truth for our inverse rendering benchmark. The method of \cite{kim2016multi} provides pseudo ground truth that, because it exploits multi-view/multi-illuminant images and known camera calibration, we expect to be considerably more accurate than the single image methods to which we do compare.
In particular, given the output from their pipeline, we perform rasterisation to generate albedo ground truth for every input image. Note that both sources of “ground truth” here are themselves only estimations, e.g.~the albedo ground truth contains ambient occlusion baked in. The colour balance of the estimated albedo is arbitrary, so we compute per-channel optimal scalings prior to computing errors. We use two metrics - MSE (mean squared error) and LMSE (local mean squared error), which are commonly used for evaluating albedo predictions. 
%The correctness of illumination predictions can be inferred by the other two, so we do not perform explicit evaluations on it. 
The quantitative evaluations are shown in Tab.~\ref{tab:am&nm_table}. For depth prediction methods, we first compute the optimal scaling onto the ground truth geometry, then compute numerical derivatives to find surface normals. These methods can only be evaluated on normal prediction. Intrinsic image methods can only be evaluated on albedo prediction. We can see that our network performs best in both albedo and normal predictions. Qualitative results can be found in Fig.~\ref{fig:benchmark}. Note that, relative to the inverse rendering methods, our result is able to explain cast shadows in the shadow map, avoiding baking them into the albedo map. As supplementary material we also show results on a video, performing inverse rendering on each frame independently.

Except the direct evaluations against the ground truth inverse rendering results, we propose two additional metrics which are reconstruction accuracy and albedo consistency. The introduced two metrics reflects how well our inverse rendering results explain the input image, and whether the physical invariants are consistent across the scene. They indirectly evaluate the correctness of our inverse rendering result. Quantitative results are shown in Tab.~\ref{tab:recon&consis} Here we measure albedo consistency by cross-projecting albedo predictions to overlapping images, followed by calculating the difference between the now-aligned albedo predictions. We show qualitative results in Fig.~\ref{fig:recon} and \ref{fig:am_consist}. Relative to \cite{yu2019inverserendernet}, our albedo maps preserve high frequency detail since we do not use a smoothness prior. Note also that we are able to extract albedo maps with consistent colour from images with very different illumination colour (Fig.~\ref{fig:am_consist}).

\begin{figure*}[!t]
\footnotesize
\centering

\begin{subfigure}[]{\textwidth}
\centering
\includegraphics[width=\textwidth, clip=true, trim=20px 65px 10px 45px]{benchmark/1.pdf}
% \caption{}
\end{subfigure}\\

\begin{subfigure}[]{\textwidth}
\centering
\includegraphics[width=\textwidth, clip=true, trim=20px 65px 10px 45px]{benchmark/2.pdf}
% \caption{}
\end{subfigure}\\

\begin{subfigure}[]{\textwidth}
\centering
\includegraphics[width=\textwidth, clip=true, trim=20px 65px 10px 45px]{benchmark/3.pdf}
% \caption{}
\end{subfigure}

    % \vspace{-0.2cm}
    \caption{Qualitative results for our inverse rendering benchmark. We show comparison against InverseRenderNet \cite{yu2019inverserendernet}, BigTime \cite{BigTimeLi18} and SIRFS \cite{BarronTPAMI2015}.}
     \label{fig:benchmark}
\end{figure*}%




\subsection{Evaluation on BigTime}\label{sec:bt_eval}

Time-lapse data allows us to use the same reconstruction and consistency metrics without the need for cross-projection. We do so on the BigTime time-lapse dataset \cite{BigTimeLi18}. The quantitative results are shown in Tab.~\ref{tab:recon&consis}. Note that the BigTime \cite{BigTimeLi18} intrinsic image method is excluded from the comparison because the method itself is entirely trained based on the dataset. All other methods including our proposed work are trained without fine-tuning on this dataset. To fairly perform the evaluations, we select 15 scenes out of nearly 200 scenes which contains large portion of indoor scenes and some outdoor scenes from the dataset. Some example results are shown in Fig.~\ref{fig:bt}. In the last example in particular, note our ability to avoid baking cast shadows into the albedo map.


\begin{figure*}[!t]
    \centering
\begingroup
\setlength{\tabcolsep}{1pt}
\renewcommand{\arraystretch}{0.7}
\small{
\resizebox{\linewidth}{!}{
\begin{tabular}{cccccccccc}
\multirow{2}{*}{\textbf{Input}}   & \multicolumn{3}{c}{\textbf{Albedo}} & \multicolumn{3}{c}{\textbf{Shading}} & \multicolumn{3}{c}{\textbf{Reconstruction}} \\
 & \cite{yu2019inverserendernet} & \cite{nestmeyer2017reflectance} & Ours &  \cite{yu2019inverserendernet} & \cite{nestmeyer2017reflectance} & Ours &  \cite{yu2019inverserendernet} & \cite{nestmeyer2017reflectance} & Ours \\
\includegraphics[width=2.cm]{bt/1/im1_sky.png}& 
\includegraphics[width=2.cm]{bt/1/am_irn1.png}&
\includegraphics[width=2.cm]{bt/1/am_filter1.png}&
\includegraphics[width=2.cm]{bt/1/am_ours1.png}&
\includegraphics[width=2.cm]{bt/1/shading_irn1.png}&
\includegraphics[width=2.cm]{bt/1/shading_filter1.png}&
\includegraphics[width=2.cm]{bt/1/shading_ours1.png}&
\includegraphics[width=2.cm]{bt/1/recon_irn1.png}&
\includegraphics[width=2.cm]{bt/1/recon_filter1.png}&
\includegraphics[width=2.cm]{bt/1/recon_ours1.png}
\\
\includegraphics[width=2.cm]{bt/1/im2_sky.png}& 
\includegraphics[width=2.cm]{bt/1/am_irn2.png}&
\includegraphics[width=2.cm]{bt/1/am_filter2.png}&
\includegraphics[width=2.cm]{bt/1/am_ours2.png}&
\includegraphics[width=2.cm]{bt/1/shading_irn2.png}&
\includegraphics[width=2.cm]{bt/1/shading_filter2.png}&
\includegraphics[width=2.cm]{bt/1/shading_ours2.png}&
\includegraphics[width=2.cm]{bt/1/recon_irn2.png}&
\includegraphics[width=2.cm]{bt/1/recon_filter2.png}&
\includegraphics[width=2.cm]{bt/1/recon_ours2.png}
\\
\vspace{-0.5mm}
\\
\includegraphics[width=2.cm]{bt/2/im1_sky.png}& 
\includegraphics[width=2.cm]{bt/2/am_irn1.png}&
\includegraphics[width=2.cm]{bt/2/am_filter1.png}&
\includegraphics[width=2.cm]{bt/2/am_ours1.png}&
\includegraphics[width=2.cm]{bt/2/shading_irn1.png}&
\includegraphics[width=2.cm]{bt/2/shading_filter1.png}&
\includegraphics[width=2.cm]{bt/2/shading_ours1.png}&
\includegraphics[width=2.cm]{bt/2/recon_irn1.png}&
\includegraphics[width=2.cm]{bt/2/recon_filter1.png}&
\includegraphics[width=2.cm]{bt/2/recon_ours1.png}
\\
\includegraphics[width=2.cm]{bt/2/im2_sky.png}& 
\includegraphics[width=2.cm]{bt/2/am_irn2.png}&
\includegraphics[width=2.cm]{bt/2/am_filter2.png}&
\includegraphics[width=2.cm]{bt/2/am_ours2.png}&
\includegraphics[width=2.cm]{bt/2/shading_irn2.png}&
\includegraphics[width=2.cm]{bt/2/shading_filter2.png}&
\includegraphics[width=2.cm]{bt/2/shading_ours2.png}&
\includegraphics[width=2.cm]{bt/2/recon_irn2.png}&
\includegraphics[width=2.cm]{bt/2/recon_filter2.png}&
\includegraphics[width=2.cm]{bt/2/recon_ours2.png}
\\
\vspace{-0.5mm}
\\
\includegraphics[width=2.cm]{bt/3/im1_sky.png}& 
\includegraphics[width=2.cm]{bt/3/am_irn1.png}&
\includegraphics[width=2.cm]{bt/3/am_filter1.png}&
\includegraphics[width=2.cm]{bt/3/am_ours1.png}&
\includegraphics[width=2.cm]{bt/3/shading_irn1.png}&
\includegraphics[width=2.cm]{bt/3/shading_filter1.png}&
\includegraphics[width=2.cm]{bt/3/shading_ours1.png}&
\includegraphics[width=2.cm]{bt/3/recon_irn1.png}&
\includegraphics[width=2.cm]{bt/3/recon_filter1.png}&
\includegraphics[width=2.cm]{bt/3/recon_ours1.png}
\\
\includegraphics[width=2.cm]{bt/3/im2_sky.png}& 
\includegraphics[width=2.cm]{bt/3/am_irn2.png}&
\includegraphics[width=2.cm]{bt/3/am_filter2.png}&
\includegraphics[width=2.cm]{bt/3/am_ours2.png}&
\includegraphics[width=2.cm]{bt/3/shading_irn2.png}&
\includegraphics[width=2.cm]{bt/3/shading_filter2.png}&
\includegraphics[width=2.cm]{bt/3/shading_ours2.png}&
\includegraphics[width=2.cm]{bt/3/recon_irn2.png}&
\includegraphics[width=2.cm]{bt/3/recon_filter2.png}&
\includegraphics[width=2.cm]{bt/3/recon_ours2.png}
\\
\vspace{-0.5mm}
\\
\includegraphics[width=2.cm]{bt/5/im1_sky.png}& 
\includegraphics[width=2.cm]{bt/5/am_irn1.png}&
\includegraphics[width=2.cm]{bt/5/am_filter1.png}&
\includegraphics[width=2.cm]{bt/5/am_ours1.png}&
\includegraphics[width=2.cm]{bt/5/shading_irn1.png}&
\includegraphics[width=2.cm]{bt/5/shading_filter1.png}&
\includegraphics[width=2.cm]{bt/5/shading_ours1.png}&
\includegraphics[width=2.cm]{bt/5/recon_irn1.png}&
\includegraphics[width=2.cm]{bt/5/recon_filter1.png}&
\includegraphics[width=2.cm]{bt/5/recon_ours1.png}
\\
\includegraphics[width=2.cm]{bt/5/im2_sky.png}& 
\includegraphics[width=2.cm]{bt/5/am_irn2.png}&
\includegraphics[width=2.cm]{bt/5/am_filter2.png}&
\includegraphics[width=2.cm]{bt/5/am_ours2.png}&
\includegraphics[width=2.cm]{bt/5/shading_irn2.png}&
\includegraphics[width=2.cm]{bt/5/shading_filter2.png}&
\includegraphics[width=2.cm]{bt/5/shading_ours2.png}&
\includegraphics[width=2.cm]{bt/5/recon_irn2.png}&
\includegraphics[width=2.cm]{bt/5/recon_filter2.png}&
\includegraphics[width=2.cm]{bt/5/recon_ours2.png}
\\
\end{tabular}
}}
\endgroup
    \caption{Qualitative results on BigTime data \cite{BigTimeLi18}. Each consecutive pair of rows shows results for two different frames from a time-lapse sequence. Col.~1: input, col.~2-4: albedo, col.~5-7: shading, col.~8-10: reconstruction. For image we show comparison between InverseRenderNet \cite{yu2019inverserendernet}, Nestmeyer \etal \cite{nestmeyer2017reflectance} and ours.}
    \label{fig:bt}
\end{figure*}






\subsection{Evaluation on IIW}\label{sec:iiw_eval}

The standard benchmark for intrinsic image decomposition is Intrinsic Images in the Wild \cite{bell14intrinsic} (IIW) which is almost exclusively indoor scenes. Since our training regime requires large multiview image datasets, we are restricted to using scene-tagged images crawled from the web, which are usually outdoors. In addition, our illumination model is learnt on outdoor, natural environments. For these reasons, we cannot perform training or fine-tuning on this indoor benchmark. However, we perform fine-tuning using two other datasets to improve performance on the intrinsic image task on indoor scenes. Specifically, we first add the BigTime dataset \cite{BigTimeLi18} for which we can compute self rendering, albedo consistency and cross-rendering losses without the need for any cross projection. Second, we sidestep the lack of surface normal labels in BigTime \cite{BigTimeLi18} by introducing the indoor normal supervisions from the NYU dataset \cite{Silberman:ECCV12}. To train our network with the mixture of all three sources of data, each training iteration is separated into two sub-steps. Firstly, the network is trained by MegaDepth \cite{MegaDepthLi18} and NYU \cite{Silberman:ECCV12} to learn indoor self-reconstruction and normal prediction. Secondly, by freezing the normal prediction decoder, only shadow and albedo prediction decoders are trained by BigTime \cite{BigTimeLi18}. We show qualitative results in Fig.~\ref{fig:iiw_figure} and quantitative results in Tab.~\ref{tab:IIW_table}. Note that our performance is very close to the state-of-the-art for methods not fine-tuned on the IIW training set, despite the fact that our inverse rendering solution is more constrained than intrinsic image methods (we must explain shading in terms of geometry, lighting and shadows whereas intrinsic image methods are free to compute arbitrary shading maps).



\begin{table}[!t]
\setlength{\tabcolsep}{7pt}
\renewcommand{\arraystretch}{1.1}
\centering
{
    \begin{tabular}{c|c|c}
    \hline
    Methods & Training data &WHDR \\ \cline{1-3}
    \hline
    \hline
    Nestmeyer \cite{nestmeyer2017reflectance} & IIW & 19.5 \\
    Li \etal \cite{li2018cgintrinsics}  & IIW & 17.5 \\ 
    % Fan \etal \cite{fan2017revisiting}  & IIW & 14.5 \\
    Sengupta \etal \cite{sengupta2019neural} & IIW & 16.7\\
    \hline
    Narihira \etal \cite{narihira2015direct}  & Sintel+MIT & 37.3 \\
    Shi \etal \cite{shi2017learning}  & ShapeNet & 59.4 \\ 
    BigTime \cite{BigTimeLi18} & BigTime & 20.3 \\
    InverseRenderNet \cite{yu2019inverserendernet}  & MegaDepth & 21.4
    \\ \cline{1-3}
    Ours  & MD+BT+NYU & 21.1
    \\ \cline{1-3}
    
    \hline
    \end{tabular}
}
\caption{Quantitative results on IIW benchmark using WHDR percentage (lower is better). The second column shows which dataset on which the networks were trained. The upper block are trained on IIW training set.}
\label{tab:IIW_table}
\end{table}



\begin{figure*}[!t]
\footnotesize
\centering
\begingroup
\setlength{\tabcolsep}{1pt}
\renewcommand{\arraystretch}{0.5}
\resizebox{\textwidth}{!}{
\begin{tabular}{cccccccc}
Input & Li \cite{BigTimeLi18} (R) & Nestmeyer \cite{nestmeyer2017reflectance} (R) & Ours (R) & Li \cite{BigTimeLi18} (S) & Nestmeyer \cite{nestmeyer2017reflectance} (S) & Ours (S) & Ours (NM) \\
\includegraphics[width=2.4cm]{iiw/iiw1/6155.png}& 
\includegraphics[width=2.4cm]{iiw/iiw1/R.png}&
\includegraphics[width=2.4cm]{iiw/iiw1/R_color.png}&
\includegraphics[width=2.4cm]{iiw/iiw1/am_our.png}&
\includegraphics[width=2.4cm]{iiw/iiw1/S.png}&
\includegraphics[width=2.4cm]{iiw/iiw1/S_color.png}&
\includegraphics[width=2.4cm]{iiw/iiw1/shading_our.png}&
\includegraphics[width=2.4cm]{iiw/iiw1/nm_our.png}\\
\includegraphics[width=2.4cm]{iiw/iiw2/10331.png}& 
\includegraphics[width=2.4cm]{iiw/iiw2/R.png}&
\includegraphics[width=2.4cm]{iiw/iiw2/R_color.png}&
\includegraphics[width=2.4cm]{iiw/iiw2/am_our.png}&
\includegraphics[width=2.4cm]{iiw/iiw2/S.png}&
\includegraphics[width=2.4cm]{iiw/iiw2/S_color.png}&
\includegraphics[width=2.4cm]{iiw/iiw2/shading_our.png}&
\includegraphics[width=2.4cm]{iiw/iiw2/nm_our.png} \\
\includegraphics[width=2.4cm]{iiw/iiw4/7005.png}& 
\includegraphics[width=2.4cm]{iiw/iiw4/R.png}&
\includegraphics[width=2.4cm]{iiw/iiw4/R_color.png}&
\includegraphics[width=2.4cm]{iiw/iiw4/am_our.png}&
\includegraphics[width=2.4cm]{iiw/iiw4/S.png}&
\includegraphics[width=2.4cm]{iiw/iiw4/S_color.png}&
\includegraphics[width=2.4cm]{iiw/iiw4/shading_our.png}&
\includegraphics[width=2.4cm]{iiw/iiw4/nm_our.png} \\
\includegraphics[width=2.4cm]{iiw/iiw5/1118.png}& 
\includegraphics[width=2.4cm]{iiw/iiw5/R.png}&
\includegraphics[width=2.4cm]{iiw/iiw5/R_color.png}&
\includegraphics[width=2.4cm]{iiw/iiw5/am_our.png}&
\includegraphics[width=2.4cm]{iiw/iiw5/S.png}&
\includegraphics[width=2.4cm]{iiw/iiw5/S_color.png}&
\includegraphics[width=2.4cm]{iiw/iiw5/shading_our.png}&
\includegraphics[width=2.4cm]{iiw/iiw5/nm_our.png} \\
\includegraphics[width=2.4cm]{iiw/iiw6/686.png}& 
\includegraphics[width=2.4cm]{iiw/iiw6/R.png}&
\includegraphics[width=2.4cm]{iiw/iiw6/R_color.png}&
\includegraphics[width=2.4cm]{iiw/iiw6/am_our.png}&
\includegraphics[width=2.4cm]{iiw/iiw6/S.png}&
\includegraphics[width=2.4cm]{iiw/iiw6/S_color.png}&
\includegraphics[width=2.4cm]{iiw/iiw6/shading_our.png}&
\includegraphics[width=2.4cm]{iiw/iiw6/nm_our.png} \\
\end{tabular}
}
\endgroup
    \vspace{-0.2cm}
    \caption{Qualitative intrinsic image results for IIW benchmark. Col.~2-4: reflectance predictions from \cite{BigTimeLi18}, \cite{nestmeyer2017reflectance} and ours. Col.~5-7: corresponding shading predictions. Col.~8: surface normal prediction for our method.}
     \label{fig:iiw_figure}
\end{figure*}%



\subsection{Evaluation on DIODE}\label{sec:diode_eval}

DIODE \cite{diode_dataset} is a very recent geometry estimation benchmark comprising images and registered depth/normal maps acquired with a Lidar scanner. The benchmark is divided into indoor and outdoor scenes and we evaluate only on the outdoor scene test set. To date, the only published result for normal map estimation is the method of Eigen \etal \cite{eigen2015predicting} against which we quantitatively compare in Tab.~\ref{tab:diode}. We show qualitative example results on this dataset in Fig.~\ref{fig:diode}.


\begin{table}[!t]
\centering
\setlength{\tabcolsep}{7pt}
\renewcommand{\arraystretch}{1}
    \begin{tabular}{c|c|c}
    \hline
    Methods & Mean & Median \\
    \hline
    \hline
    Eigen \etal \cite{eigen2015predicting}  & 31.9 & 24.7 \\
    ours  & \textbf{23.9} & \textbf{15.5} \\ % 22-38
    \hline
    \end{tabular}
\caption{Quantitative surface normal prediction errors on the DIODE dataset \cite{diode_dataset}. We show mean and median angular errors in degrees for the outdoor test set.}
\label{tab:diode}
\end{table}


\begin{figure}[!t]
    \centering
\begingroup
\setlength{\tabcolsep}{1pt}
\renewcommand{\arraystretch}{0.5}
\small{
\begin{tabular}{ccc}
Input & GT Normal & {\footnotesize Estimated Normal} \\

\includegraphics[width=2.8cm]{diode/2/im.png}& 
\includegraphics[width=2.8cm]{diode/2/nm_gt.png}&
\includegraphics[width=2.8cm]{diode/2/nm_pred.png}
\\
\includegraphics[width=2.8cm]{diode/1/im.png}& 
\includegraphics[width=2.8cm]{diode/1/nm_gt.png}&
\includegraphics[width=2.8cm]{diode/1/nm_pred.png}
\\
\includegraphics[width=2.8cm]{diode/3/im.png}& 
\includegraphics[width=2.8cm]{diode/3/nm_gt.png}&
\includegraphics[width=2.8cm]{diode/3/nm_pred.png}
\\
\includegraphics[width=2.8cm]{diode/4/im.png}& 
\includegraphics[width=2.8cm]{diode/4/nm_gt.png}&
\includegraphics[width=2.8cm]{diode/4/nm_pred.png}
\\
\end{tabular}
}
\endgroup
    \caption{Qualitative results for surface normal prediction on DIODE dataset \cite{diode_dataset}. Left to right: input, ground truth normal map, estimated normal map.}
    \label{fig:diode}
\end{figure}


\subsection{Illumination estimation}\label{sec:illu_eval}
The benchmark dataset in \cite{yu20relightNet} consists of images capturing a monument under six different illumination conditions from multiple viewpoints. Along with this multi-view and multi-illumination image collection, the six HDR illumination environment maps and necessary alignments between image and environment map pairs are also provided. In order to evaluate our performance on lighting estimation, we feed each image into our network and compare our lighting prediction with ground truth. Since our network can only infer the lighting represented by spherical harmonics, we project the ground truth environment map onto order 2 spherical harmonics. To ensure a fair evaluation the comparison regarding backside lighting is omitted, so we compare only the front side by measuring the distance between hemispheres lit by our lighting prediction and by ground truth spherical harmonic lighting. We show quantitative evaluations in Tab.~\ref{tab:illu_eval} and qualitative evaluations in Fig.~\ref{fig:illu_eval}. Since there is an overall scale ambiguity between albedo and lighting, in quantitative evaluations, two scaling methods are applied. The global scaling metric is computed by applying an optimal global intensity scale prior to measuring errors, and per-colour scaling method is conducted by applying an optimal scaling to each colour channel. Accordingly, the qualitative results for these two scaling methods are shown in Fig.~\ref{fig:illu_eval}.


\begin{figure*}[!t]
    \centering
\begingroup
\setlength{\tabcolsep}{1pt}
\renewcommand{\arraystretch}{0.5}
\resizebox{\linewidth}{!}{
\begin{tabular}{ccccccc}
Input & Environment map & GT illu. & SIRFS \cite{BarronTPAMI2015} & {\footnotesize InverseRenderNet \cite{yu2019inverserendernet}} & Ours \\
\includegraphics[height=2.8cm]{illu_eval/5/1_10_4.png}& 
\includegraphics[height=2.8cm]{illu_eval/5/em.png}&
\includegraphics[height=2.8cm]{illu_eval/5/gt_lighting_recon.png}&
\includegraphics[height=2.8cm]{illu_eval/5/sirfs_illu_rgbScale.png}&
\includegraphics[height=2.8cm]{illu_eval/5/irn_illu_rgbScale.png}&
\includegraphics[height=2.8cm]{illu_eval/5/ours_illu_rgbScale.png}
\\
\includegraphics[height=2.8cm]{illu_eval/3/5_7_3.png}&
\includegraphics[height=2.8cm]{illu_eval/3/em.png}&
\includegraphics[height=2.8cm]{illu_eval/3/gt_lighting_recon.png}&
\includegraphics[height=2.8cm]{illu_eval/3/sirfs_illu_rgbScale.png}&
\includegraphics[height=2.8cm]{illu_eval/3/irn_illu_rgbScale.png}&
\includegraphics[height=2.8cm]{illu_eval/3/ours_illu_rgbScale.png}
\\
\includegraphics[height=2.8cm]{illu_eval/1/2_11_5.png}& 
\includegraphics[height=2.8cm]{illu_eval/1/em.png}&
\includegraphics[height=2.8cm]{illu_eval/1/gt_lighting_recon.png}&
\includegraphics[height=2.8cm]{illu_eval/1/sirfs_illu_globalScale.png}&
\includegraphics[height=2.8cm]{illu_eval/1/irn_illu_globalScale.png}&
\includegraphics[height=2.8cm]{illu_eval/1/ours_illu_globalScale.png}
\\
\includegraphics[height=2.8cm]{illu_eval/4/6_8_3.png}& 
\includegraphics[height=2.8cm]{illu_eval/4/em.png}&
\includegraphics[height=2.8cm]{illu_eval/4/gt_lighting_recon.png}&
\includegraphics[height=2.8cm]{illu_eval/4/sirfs_illu_globalScale.png}&
\includegraphics[height=2.8cm]{illu_eval/4/irn_illu_globalScale.png}&
\includegraphics[height=2.8cm]{illu_eval/4/ours_illu_globalScale.png}
\\
\end{tabular}
}
\endgroup
    \caption{Qualitative evaluation for illumination estimation. Captured environment maps are shown in col. 2, which are projected to spherical harmonics and used for relighting hemisphere to generate ground truth illumination estimations as demonstrated in col. 3. Illumination estimates from SIRFS \cite{BarronTPAMI2015}, InverseRenderNet \cite{yu2019inverserendernet} and our proposed method are shown in cols 4-6. The first two rows are per-colour scaled, and last two are global scaled.}
    \label{fig:illu_eval}
\end{figure*}

\begin{table}[!t]
\centering
\setlength{\tabcolsep}{7pt}
\renewcommand{\arraystretch}{1}
    \begin{tabular}{c|c|c}
    \hline
    Methods & Global scale & Per-colour scale \\
    \hline
    \hline
    SIRFS \cite{BarronTPAMI2015}  & 0.100 & 0.089 \\
    InverseRenderNet \cite{yu2019inverserendernet}  & 0.050 & 0.041 \\
    Ours  & \textbf{0.038} & \textbf{0.033} \\ % 22-38
    \hline
    \end{tabular}
\caption{Quantitative results for illumination estimation. We show global-scale and per-colour-scale MSE errors.}
\label{tab:illu_eval}
\end{table}



\subsection{Ablation study}
In this section, we seek insight on some key design choices by comparing the performance of our proposed network and ablated models on evaluation datasets used above. We report the comparison between our full model and ablated model trained without VGG loss, without cross-projection loss and without cross-rendering loss. The quantitative results are summarised in Tab.~\ref{tab:ablation_study}. Our proposed model outperforms the ablated methods on all metrics across all datasets. Fig.~\ref{fig:ablation_study} visualises the qualitative comparison on different training setups. Here, without cross-projection loss the network cannot remove shadows from albedo prediction. Without cross-rendering loss the network can only predict over-smoothed albedo. Without VGG loss, albedo prediction looks `bleached', since shading and shadow predictions capture too much darkness - shading predictions of this ablated variant shows less brightness contrast and darker intensity than that of our full model. While training without VGG loss results in larger errors, we found that finetuning the weights $w_1$, $w_3$ and $w_4$ of losses for specific tasks leads to better performance than our full model. However, it cannot providing consistently the best performance across all metrics with a uniform weights setting.

\begin{table}[!t]
\centering
\resizebox{\columnwidth}{!}{%
\setlength{\tabcolsep}{7pt}
\renewcommand{\arraystretch}{1}
    \begin{tabular}{c|c|c|c}
    \hline
    % Methods & \shortstack{MD reflectance \\ (MSE)} & IIW & \shortstack{Illumination \\ (global scale)} \\
    \multirow{2}{*}{Methods} & MD reflectance & \multirow{2}{*}{IIW} & Illumination \\
    & (MSE) & & (global scale) \\
    \hline
    \hline
    Full & \textbf{0.0093} & \textbf{21.1} & \textbf{0.038} \\
    w/o VGG & 0.0095 & 21.6 & 0.053 \\
    w/o cross-projection loss & 0.0153 & 21.6 & 0.052\\
    w/o cross-rendering loss  & 0.0101 & 21.4 & 0.052\\ % 22-38
    \hline
    \end{tabular}
}%
\caption{Quantitative comparison for ablation study. Here, we select some representative metrics used in Sec. \ref{sec:md_eval}, \ref{sec:iiw_eval} and \ref{sec:illu_eval}, which are MSE-based reflectance errors on MegaDepth test data (col. 1), IIW benchmark score (col. 2), and global-scaled Illumination estimation errors (col. 3).}
\label{tab:ablation_study}
\end{table}



\begin{figure}[!t]
    \centering
\begingroup
\setlength{\tabcolsep}{1pt}
\renewcommand{\arraystretch}{0.5}
% \small{
\resizebox{\linewidth}{!}{
\begin{tabular}{cccc}
 \multicolumn{4}{c}{Input}\\
 \multicolumn{4}{c}{\includegraphics[width=2.8cm]{ablation_study/img.png}}\\
\includegraphics[width=2.8cm]{ablation_study/full/albedo.png}&
\includegraphics[width=2.8cm]{ablation_study/no_vgg/albedo.png}&
\includegraphics[width=2.8cm]{ablation_study/no_amconsist/albedo.png}&
\includegraphics[width=2.8cm]{ablation_study/no_crossrender/albedo.png}
\\
% Normal&
\includegraphics[width=2.8cm]{ablation_study/full/nm_pred.png}&
\includegraphics[width=2.8cm]{ablation_study/no_vgg/nm_pred.png}&
\includegraphics[width=2.8cm]{ablation_study/no_amconsist/nm_pred.png}&
\includegraphics[width=2.8cm]{ablation_study/no_crossrender/nm_pred.png}
\\
% Shading&
\includegraphics[width=2.8cm]{ablation_study/full/shadingWithSd.png}&
\includegraphics[width=2.8cm]{ablation_study/no_vgg/shadingWithSd.png}&
\includegraphics[width=2.8cm]{ablation_study/no_amconsist/shadingWithSd.png}&
\includegraphics[width=2.8cm]{ablation_study/no_crossrender/shadingWithSd.png}
\\
 \multirow{2}{*}{Full model} & \multirow{2}{*}{w/o VGG} & w/o cross- & w/o cross- \\
 & & projection & rendering \\
\end{tabular}
% }
}
\endgroup
    \caption{Qualitative comparison for ablation study. Row 2: albedo, row 3: surface normals, row 4: shading. Ablation conditions are shown column-wise.}
    \label{fig:ablation_study}
\end{figure}


\section{Limitations and failure cases}
We present illustrative failure cases in Fig.~\ref{fig:failures}. In the first row, the dark plaque in the centre is mainly explained by shadowing rather than dark albedo. Meanwhile the actual cast shadow, while present in the shadow map, still corrupts the albedo. The second row illustrates poor performance in regions that lack MVS supervision. In particular, the estimation of shadows cast on the ground plane are usually worse compared with shadows on buildings. Similarly, MVS reconstructions fail on dynamic regions containing people and so our result tends to treat pedestrians as planar surfaces (see the third row). In addition, Fig.~\ref{fig:illu_eval} highlights the remaining colour ambiguity between illumination and albedo. As shown in the globally scaled results, our result always yields a biased prediction in terms of the colour, although the intensity distribution is plausible.


\begin{figure}[!t]
    \centering
\begingroup
\setlength{\tabcolsep}{1pt}
\renewcommand{\arraystretch}{0.5}
% \small{
\resizebox{\linewidth}{!}{
\begin{tabular}{ccccc}
Input & Albedo & Normal & Shading & Shadow \\

\includegraphics[width=2.8cm]{failure_cases/1/img.png}& 
\includegraphics[width=2.8cm]{failure_cases/1/albedo.png}&
\includegraphics[width=2.8cm]{failure_cases/1/nm_pred.png}&
\includegraphics[width=2.8cm]{failure_cases/1/shading.png}&
\includegraphics[width=2.8cm]{failure_cases/1/shadow.png}
\\
\includegraphics[width=2.8cm]{failure_cases/2/img.png}& 
\includegraphics[width=2.8cm]{failure_cases/2/albedo.png}&
\includegraphics[width=2.8cm]{failure_cases/2/nm_pred.png}&
\includegraphics[width=2.8cm]{failure_cases/2/shading.png}&
\includegraphics[width=2.8cm]{failure_cases/2/shadow.png}
\\
\includegraphics[width=2.8cm]{failure_cases/3/img.png}& 
\includegraphics[width=2.8cm]{failure_cases/3/albedo.png}&
\includegraphics[width=2.8cm]{failure_cases/3/nm_pred.png}&
\includegraphics[width=2.8cm]{failure_cases/3/shading.png}&
\includegraphics[width=2.8cm]{failure_cases/3/shadow.png}
\\
\end{tabular}
% }
}
\endgroup
    \caption{Failure cases. Left to right: input, albedo estimate, normal estimate, shading and shadow estimate.}
    \label{fig:failures}
\end{figure}



\section{Conclusions}

We have shown for the first time that the task of inverse rendering can be learnt from real world images in uncontrolled conditions. Our results show that ``shape-from-shading'' in the wild is possible and our results are far superior to classical methods. We are also competitive against state-of-the-art methods trained specifically for related subtasks. It is interesting to ponder how this feat is achieved. We believe the reason this is possible is because of the large range of cues that the deep network can exploit, for example shading, texture, ambient occlusion, perhaps even high level semantic concepts learnt from the diverse data. For example, once a region is recognised as a ``window'', the possible shape and configuration is much restricted. Recognising a scene as a man-made building suggests the presence of many parallel and orthogonal planes. These sort of cues would be extremely difficult to exploit in hand-crafted solutions.

An interesting question is whether our network learns general principles of shape-from-shading that generalise beyond the types of outdoor, primarily urban scenes on which the network was trained. Another question is whether the network's perceptions and resolutions of ambiguities are similar to that of a human. We show some qualitative examples to suggest that both of these are indeed the case. We begin in Fig.~\ref{fig:plates} by showing shape estimation results on the plates shading illusion. We note initially that our network is able to estimate meaningful shape from images that are completely unlike the training data. The plates are all the right side up, i.e.~they are convex. The illusion was originally presented in the orientation shown in (c), where lighting is from the side. With no prior for a particular direction for side lighting, human viewers flip between interpreting the plates as convex or concave. In (a) and (b), the human preference for lighting from above \cite{kleffner1992perception} is shared by our network which interprets the plates as all convex (a) and all concave (b), estimating top lighting in both cases. For (c) and (d), although our lighting model can capture side lighting (Fig.~\ref{fig:illu_model}, first component), because the scene is locally ambiguous it is resolved differently for different parts of the image (mix of convex and concave interpretations) and the least squares lighting estimate is an average of the two interpretations, resulting in an approximately top down lighting estimate.

\begin{figure}
    \centering
    \includegraphics[width=0.9\linewidth]{illusions/plates.pdf}
    \caption{Shape estimation results on the plates shading illusion. For four orientations of the image (a-d) we show the input image and the estimated normal map with estimated lighting inset. }
    \label{fig:plates}
\end{figure}

In Fig.~\ref{fig:blade} we show shape estimation results on the turbine blade illusion \cite{mather2017turbine}. Here, the convexity prior preferred by humans interprets the S-shaped cross section as a cylinder leading to an inconsistent lighting condition relative to the rest of the scene. This makes the blade look fake, as if it has been added afterwards. In this case, our network does not seem to have learnt such a strong convexity prior and correctly reconstructs the S-shaped profile. This can partly be explained by the fact that our network is forced to reconstruct a single, global illumination condition which differs from the human interpretation of this scene.

\begin{figure}
    \centering
    \includegraphics[width=0.48\linewidth]{illusions/blade2.jpg}
    \includegraphics[width=0.48\linewidth]{illusions/blade2_nm_pred.png}
    \caption{Shape estimation on the turbine blade illusion \cite{mather2017turbine}.}
    \label{fig:blade}
\end{figure}

In Fig.~\ref{fig:trompe} we show two example results on Trompe-l'{\oe}il illusions and one for a mirror illusion. Trompe-l'{\oe}il use albedo variations (i.e.~painted surface texture) to depict 3D scenes, giving the illusion of additional geometric variation. A mirror illusion is similar except that the albedo variation is a reflection of another part of the real scene. In all examples, the shape reconstruction is fooled by the illusion. In the example in the first column, both the sky segmentation network and our inverse rendering network are fooled. The shape of the painted windows, doors and fountain as well as the false building corners are recovered. For the reflection illusion in the third column, the shape of the reflected scene is reconstructed. This is not surprising since we do not model reflections and the MVS supervision would not correctly deal with reflective surfaces.

\begin{figure}
    \centering
    \begingroup
\setlength{\tabcolsep}{1pt}
\renewcommand{\arraystretch}{0.5}
    \resizebox{\linewidth}{!}{
\begin{tabular}{ccc}
    \includegraphics[height=1.5cm]{illusions/building.jpg}&
    \includegraphics[height=1.5cm]{illusions/louvre.jpg}&
    \includegraphics[height=1.5cm]{illusions/mirror.jpg}\\
    \includegraphics[height=1.5cm]{illusions/building_nm_pred.png}&
    \includegraphics[height=1.5cm]{illusions/louvre_nm_pred.png}&
    \includegraphics[height=1.5cm]{illusions/mirror_nm_pred.png}
    \end{tabular}}
    \endgroup
    \caption{Shape estimation in the presence of Trompe-l'{\oe}il and mirror illusions.}
    \label{fig:trompe}
\end{figure}

There are many promising ways in which this work can be extended. First, our modelling assumptions could be relaxed, for example using more general reflectance models and estimating global illumination effects. Second, our network could be combined with a depth prediction network. Either the two networks could be applied independently and then the depth and normal maps merged \cite{yu2019depth}, or a unified network could be trained in which the normals computed from the depth map are used to compute the losses we use in this paper. Networks that explicitly reason about scene layout \cite{zou2018layoutnet} could be used to directly supervise ground plane predictions. Third, our network could benefit from losses used in training intrinsic image decomposition networks \cite{BigTimeLi18}. Fourth, our lighting prior could be extended to better handle indoor scenes. Finally, our fixed reflectance and illumination model could be made partially learnable in order to be able to better explain real world appearance \cite{sengupta2019neural}. Our code, trained model and inverse rendering benchmark data is available at https://github.com/YeeU/InverseRenderNet.

\section*{Acknowledgment}

W.~Smith was supported by the Royal Academy of Engineering under the Leverhulme Trust Senior Fellowship scheme. The Titan Xp used for this research was donated by the NVIDIA Corporation.

\bibliographystyle{IEEEtran}
\bibliography{refs}

%\newpage
\vspace{-0.7cm}

\begin{IEEEbiography}[{\includegraphics[width=1in,clip,keepaspectratio]{Ye.jpg}}]{Ye Yu}
received the B.Sc. degree in Computer Science from University of Science and Technology Beijing, China in 2010 and the M.Sc. degree in Visual Information Processing from the Imperial College London, UK in 2011. He is currently pursuing the Ph.D. degree at the University of York where he is a member of the Computer Vision and Pattern Recognition research group. His research interests include inverse rendering and image rendering. 
\end{IEEEbiography}

\vspace{-0.5cm}

\begin{IEEEbiography}[{\includegraphics[width=1in,clip,keepaspectratio]{smith.jpg}}]{William Smith}
received the BSc degree in computer science, and the PhD degree 
in computer vision from the University of York, York, United Kingdom. He is currently a Reader with the Department of Computer Science, University of York, York, United Kingdom. He holds a Royal Academy of Engineering/The Leverhulme Trust Senior Research Fellowship. His research interests are in shape and appearance modelling, model-based supervision and physics-based vision. He has published more than 100 papers in international conferences and journals.
\end{IEEEbiography}


\end{document}

main.bbl
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
  \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl

\bibitem{BarronTPAMI2015}
J.~T. Barron and J.~Malik, ``Shape, illumination, and reflectance from
  shading,'' \emph{{IEEE} Trans. Pattern Anal. Mach. Intell.}, 2015.

\bibitem{Langguth:12}
F.~Langguth, ``Photometric stereo for outdoor webcams,'' in \emph{Proc. IEEE
  Conference on Computer Vision and Pattern Recognition (CVPR)}, 2012, pp.
  262--269.

\bibitem{aldrian2013inverse}
O.~Aldrian and W.~Smith, ``Inverse rendering of faces with a 3d morphable
  model.'' \emph{{IEEE} Trans. Pattern Anal. Mach. Intell.}, vol.~35, no.~5,
  pp. 1080--1093, 2013.

\bibitem{Snavely:08}
M.~Goesele, N.~Snavely, B.~Curless, H.~Hoppe, and S.~M. Seitz, ``Multi-view
  stereo for community photo collections,'' \emph{Proc. International
  Conference on Computer Vision (ICCV)}, pp. 1--8, 2007.

\bibitem{Furukawa:10}
Y.~Furukawa and J.~Ponce, ``Accurate, dense, and robust multiview stereopsis,''
  \emph{{IEEE} Trans. Pattern Anal. Mach. Intell.}, vol.~32, no.~8, pp.
  1362--1376, 2010.

\bibitem{eigen2014depth}
D.~Eigen, C.~Puhrsch, and R.~Fergus, ``Depth map prediction from a single image
  using a multi-scale deep network,'' in \emph{Proc. Advances in neural
  information processing systems (NIPS)}, 2014, pp. 2366--2374.

\bibitem{eigen2015predicting}
D.~Eigen and R.~Fergus, ``Predicting depth, surface normals and semantic labels
  with a common multi-scale convolutional architecture,'' in \emph{Proc.
  International Conference on Computer Vision (ICCV)}, 2015, pp. 2650--2658.

\bibitem{lettry2018darn}
L.~Lettry, K.~Vanhoey, and L.~Van~Gool, ``{DARN}: a deep adversarial residual
  network for intrinsic image decomposition,'' in \emph{Proc. IEEE Winter
  Conference on Applications of Computer Vision (WACV)}, 2018, pp. 1359--1367.

\bibitem{tewari2017mofa}
A.~Tewari, M.~Zollh{\"o}fer, H.~Kim, P.~Garrido, F.~Bernard, P.~P{\'e}rez, and
  C.~Theobalt, ``Mofa: Model-based deep convolutional face autoencoder for
  unsupervised monocular reconstruction,'' in \emph{Proc. International
  Conference on Computer Vision (ICCV)}, vol.~2, no.~3, 2017, p.~5.

\bibitem{zheng2018t2net}
C.~Zheng, T.-J. Cham, and J.~Cai, ``T2net: Synthetic-to-realistic translation
  for solving single-image depth estimation tasks,'' in \emph{Proc. European
  Conference on Computer Vision (ECCV)}, 2018, pp. 767--783.

\bibitem{yu2019inverserendernet}
Y.~Yu and W.~A. Smith, ``{InverseRenderNet}: Learning single image inverse
  rendering,'' in \emph{Proceedings of the IEEE/CVF Conference on Computer
  Vision and Pattern Recognition (CVPR)}, 2019.

\bibitem{Kazhdan:13}
M.~Kazhdan and H.~Hoppe, ``Screened {P}oisson surface reconstruction,''
  \emph{{ACM} Trans. Graph.}, vol.~32, no.~3, pp. 29:1--29:13, 2013.

\bibitem{Alldrin:08}
N.~Alldrin, T.~Zickler, and D.~Kriegman, ``Photometric stereo with
  non-parametric and spatially-varying reflectance,'' in \emph{Proc. IEEE
  Conference on Computer Vision and Pattern Recognition (CVPR)}, 2008.

\bibitem{Goldman:09}
D.~B. Goldman, B.~Curless, A.~Hertzmann, and S.~M. Seitz, ``Shape and
  spatially-varying brdfs from photometric stereo,'' \emph{{IEEE} Trans.
  Pattern Anal. Mach. Intell.}, vol.~32, no.~6, pp. 1060--1071, 2010.

\bibitem{Haber:09}
T.~Haber, C.~Fuchs, P.~Bekaer, H.~P. Seidel, M.~Goesele, and H.~P.~A. Lensch,
  ``Relighting objects from image collections,'' in \emph{Proc. IEEE Conference
  on Computer Vision and Pattern Recognition (CVPR)}, 2009, pp. 627--634.

\bibitem{kim2016multi}
K.~Kim, A.~Torii, and M.~Okutomi, ``Multi-view inverse rendering under
  arbitrary illumination and albedo,'' in \emph{Proc. European Conference on
  Computer Vision (ECCV)}, 2016, pp. 750--767.

\bibitem{jeon2014intrinsic}
J.~Jeon, S.~Cho, X.~Tong, and S.~Lee, ``Intrinsic image decomposition using
  structure-texture separation and surface normals,'' in \emph{Proc. European
  Conference on Computer Vision (ECCV)}, 2014, pp. 218--233.

\bibitem{romeiro2008passive}
F.~Romeiro, Y.~Vasilyev, and T.~Zickler, ``Passive reflectometry,'' in
  \emph{Proc. European Conference on Computer Vision (ECCV)}.\hskip 1em plus
  0.5em minus 0.4em\relax Springer, 2008, pp. 859--872.

\bibitem{lombardi2015reflectance}
S.~Lombardi and K.~Nishino, ``Reflectance and illumination recovery in the
  wild,'' \emph{{IEEE} Trans. Pattern Anal. Mach. Intell.}, vol.~38, no.~1, pp.
  129--141, 2015.

\bibitem{oxholm2015shape}
G.~Oxholm and K.~Nishino, ``Shape and reflectance estimation in the wild,''
  \emph{{IEEE} Trans. Pattern Anal. Mach. Intell.}, vol.~38, no.~2, pp.
  376--389, 2015.

\bibitem{lombardi2016radiometric}
S.~Lombardi and K.~Nishino, ``Radiometric scene decomposition: Scene
  reflectance, illumination, and geometry from rgb-d images,'' in \emph{Proc.
  International Conference on 3D Vision (3DV)}.\hskip 1em plus 0.5em minus
  0.4em\relax IEEE, 2016, pp. 305--313.

\bibitem{romeiro2010blind}
F.~Romeiro and T.~Zickler, ``Blind reflectometry,'' in \emph{Proc. European
  Conference on Computer Vision (ECCV)}.\hskip 1em plus 0.5em minus 0.4em\relax
  Springer, 2010, pp. 45--58.

\bibitem{laina2016deeper}
I.~Laina, C.~Rupprecht, V.~Belagiannis, F.~Tombari, and N.~Navab, ``Deeper
  depth prediction with fully convolutional residual networks,'' in \emph{Proc.
  International Conference on 3D Vision (3DV)}, 2016, pp. 239--248.

\bibitem{wang2015towards}
P.~Wang, X.~Shen, Z.~Lin, S.~Cohen, B.~Price, and A.~L. Yuille, ``Towards
  unified depth and semantic prediction from a single image,'' in \emph{Proc.
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, 2015, pp.
  2800--2809.

\bibitem{liu2015deep}
F.~Liu, C.~Shen, and G.~Lin, ``Deep convolutional neural fields for depth
  estimation from a single image,'' in \emph{Proc. IEEE Conference on Computer
  Vision and Pattern Recognition (CVPR)}, 2015, pp. 5162--5170.

\bibitem{xu2017multi}
D.~Xu, E.~Ricci, W.~Ouyang, X.~Wang, and N.~Sebe, ``Multi-scale continuous
  {CRF}s as sequential deep networks for monocular depth estimation,'' in
  \emph{Proc. IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR)}, vol.~1, 2017.

\bibitem{fu2018deep}
H.~Fu, M.~Gong, C.~Wang, K.~Batmanghelich, and D.~Tao, ``Deep ordinal
  regression network for monocular depth estimation,'' in \emph{Proc. IEEE
  Conference on Computer Vision and Pattern Recognition (CVPR)}, 2018, pp.
  2002--2011.

\bibitem{MegaDepthLi18}
Z.~Li and N.~Snavely, ``Megadepth: Learning single-view depth prediction from
  internet photos,'' in \emph{Proc. IEEE Conference on Computer Vision and
  Pattern Recognition (CVPR)}, 2018.

\bibitem{chen2016single}
W.~Chen, Z.~Fu, D.~Yang, and J.~Deng, ``Single-image depth perception in the
  wild,'' in \emph{Proc. Advances in neural information processing systems
  (NIPS)}, 2016, pp. 730--738.

\bibitem{kendall2017end}
A.~Kendall, H.~Martirosyan, S.~Dasgupta, P.~Henry, R.~Kennedy, A.~Bachrach, and
  A.~Bry, ``End-to-end learning of geometry and context for deep stereo
  regression,'' in \emph{Proc. International Conference on Computer Vision
  (ICCV)}, 2017, pp. 66--75.

\bibitem{garg2016unsupervised}
R.~Garg, V.~K. BG, G.~Carneiro, and I.~Reid, ``Unsupervised {CNN} for single
  view depth estimation: Geometry to the rescue,'' in \emph{Proc. European
  Conference on Computer Vision (ECCV)}, 2016, pp. 740--756.

\bibitem{godard2017unsupervised}
C.~Godard, O.~Mac~Aodha, and G.~J. Brostow, ``Unsupervised monocular depth
  estimation with left-right consistency,'' in \emph{Proc. IEEE Conference on
  Computer Vision and Pattern Recognition (CVPR)}, 2017.

\bibitem{zhou2017unsupervised}
T.~Zhou, M.~Brown, N.~Snavely, and D.~G. Lowe, ``Unsupervised learning of depth
  and ego-motion from video,'' in \emph{Proc. IEEE Conference on Computer
  Vision and Pattern Recognition (CVPR)}, vol.~2, no.~6, 2017, p.~7.

\bibitem{vijayanarasimhan2017sfm}
S.~Vijayanarasimhan, S.~Ricco, C.~Schmid, R.~Sukthankar, and K.~Fragkiadaki,
  ``{SfM-net}: Learning of structure and motion from video,'' \emph{arXiv
  preprint arXiv:1704.07804}, 2017.

\bibitem{wang2018learning}
C.~Wang, J.~M. Buenaposada, R.~Zhu, and S.~Lucey, ``Learning depth from
  monocular videos using direct methods,'' in \emph{Proc. IEEE Conference on
  Computer Vision and Pattern Recognition (CVPR)}, 2018, pp. 2022--2030.

\bibitem{monodepth2}
C.~Godard, O.~{Mac Aodha}, M.~Firman, and G.~J. Brostow, ``Digging into
  self-supervised monocular depth prediction,'' in \emph{Proc. International
  Conference on Computer Vision (ICCV)}, October 2019.

\bibitem{tulsiani2017multi}
S.~Tulsiani, T.~Zhou, A.~A. Efros, and J.~Malik, ``Multi-view supervision for
  single-view reconstruction via differentiable ray consistency,'' in
  \emph{Proc. IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR)}, vol.~1, no.~2, 2017, p.~3.

\bibitem{ji2017surfacenet}
M.~Ji, J.~Gall, H.~Zheng, Y.~Liu, and L.~Fang, ``{SurfaceNet}: an end-to-end 3d
  neural network for multiview stereopsis,'' in \emph{Proc. International
  Conference on Computer Vision (ICCV)}, 2017, pp. 2307--2315.

\bibitem{narihira2015direct}
T.~Narihira, M.~Maire, and S.~X. Yu, ``Direct intrinsics: Learning
  albedo-shading decomposition by convolutional regression,'' in \emph{Proc.
  International Conference on Computer Vision (ICCV)}, 2015, pp. 2992--2992.

\bibitem{han2018learning}
G.~Han, X.~Xie, J.~Lai, and W.-S. Zheng, ``Learning an intrinsic image
  decomposer using synthesized rgb-d dataset,'' \emph{IEEE Signal Processing
  Letters}, vol.~25, no.~6, pp. 753--757, 2018.

\bibitem{e.20181172}
S.~Bi, N.~K. Kalantari, and R.~Ramamoorthi, ``{Deep Hybrid Real and Synthetic
  Training for Intrinsic Decomposition},'' in \emph{Eurographics Symposium on
  Rendering - Experimental Ideas \& Implementations}, W.~Jakob and
  T.~Hachisuka, Eds.\hskip 1em plus 0.5em minus 0.4em\relax The Eurographics
  Association, 2018.

\bibitem{fan2017revisiting}
Q.~Fan, J.~Yang, G.~Hua, B.~Chen, and D.~Wipf, ``Revisiting deep intrinsic
  image decompositions,'' in \emph{Proc. IEEE Conference on Computer Vision and
  Pattern Recognition (CVPR)}, 2018, pp. 8944--8952.

\bibitem{BigTimeLi18}
Z.~Li and N.~Snavely, ``Learning intrinsic image decomposition from watching
  the world,'' in \emph{Proc. IEEE Conference on Computer Vision and Pattern
  Recognition (CVPR)}, 2018.

\bibitem{li2018cgintrinsics}
------, ``{CGIntrinsics}: Better intrinsic image decomposition through
  physically-based rendering,'' in \emph{Proc. European Conference on Computer
  Vision (ECCV)}, 2018.

\bibitem{ma2018single}
W.-C. Ma, H.~Chu, B.~Zhou, R.~Urtasun, and A.~Torralba, ``Single image
  intrinsic decomposition without a single intrinsic image,'' in \emph{Proc.
  European Conference on Computer Vision (ECCV)}, 2018, pp. 201--217.

\bibitem{Baslamisli_2018_CVPR}
A.~S. Baslamisli, H.-A. Le, and T.~Gevers, ``{CNN} based learning using
  reflection and retinex models for intrinsic image decomposition,'' in
  \emph{Proc. IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR)}, June 2018.

\bibitem{shelhamer2015scene}
E.~Shelhamer, J.~T. Barron, and T.~Darrell, ``Scene intrinsics and depth from a
  single image,'' in \emph{Proc. IEEE Conference on Computer Vision and Pattern
  Recognition (CVPR) Workshops}, 2015, pp. 37--44.

\bibitem{shu2017neural}
Z.~Shu, E.~Yumer, S.~Hadap, K.~Sunkavalli, E.~Shechtman, and D.~Samaras,
  ``Neural face editing with intrinsic image disentangling,'' in \emph{Proc.
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, 2017, pp.
  5444--5453.

\bibitem{sengupta2017sfsnet}
S.~Sengupta, A.~Kanazawa, C.~D. Castillo, and D.~W. Jacobs, ``{SfSNet}:
  Learning shape, reflectance and illuminance of faces `in the wild','' in
  \emph{Proc. IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR)}, 2018.

\bibitem{nestmeyer2019structural}
T.~Nestmeyer, I.~Matthews, J.-F. Lalonde, and A.~M. Lehrmann, ``Structural
  decompositions for end-to-end relighting,'' \emph{arXiv preprint
  arXiv:1906.03355}, 2019.

\bibitem{kanamori2018relighting}
Y.~Kanamori and Y.~Endo, ``Relighting humans: occlusion-aware inverse rendering
  for full-body human images,'' \emph{{ACM} Trans. Graph. (Proceedings of
  {SIGGRAPH} Asia)}, vol.~37, no.~6, 2018.

\bibitem{aittala2016reflectance}
M.~Aittala, T.~Aila, and J.~Lehtinen, ``Reflectance modeling by neural texture
  synthesis,'' \emph{{ACM} Trans. Graph.}, vol.~35, no.~4, p.~65, 2016.

\bibitem{Gao:2019:DIR}
D.~Gao, X.~Li, Y.~Dong, P.~Peers, K.~Xu, and X.~Tong, ``Deep inverse rendering
  for high resolution {SVBRDF} estimation from an arbitrary number of images,''
  \emph{{ACM} Trans. Graph.}, vol.~37, no.~4, July 2019.

\bibitem{li2017modeling}
X.~Li, Y.~Dong, P.~Peers, and X.~Tong, ``Modeling surface appearance from a
  single photograph using self-augmented convolutional neural networks,''
  \emph{{ACM} Trans. Graph.}, vol.~36, no.~4, p.~45, 2017.

\bibitem{PGZED19}
\BIBentryALTinterwordspacing
J.~Philip, M.~Gharbi, T.~Zhou, A.~Efros, and G.~Drettakis, ``Multi-view
  relighting using a geometry-aware network,'' \emph{{ACM} Trans. Graph.
  (Proceedings of {SIGGRAPH})}, vol.~38, no.~4, July 2019. [Online]. Available:
  \url{http://www-sop.inria.fr/reves/Basilic/2019/PGZED19}
\BIBentrySTDinterwordspacing

\bibitem{kulkarni2015deep}
T.~D. Kulkarni, W.~F. Whitney, P.~Kohli, and J.~Tenenbaum, ``Deep convolutional
  inverse graphics network,'' in \emph{Proc. Advances in neural information
  processing systems (NIPS)}, 2015, pp. 2539--2547.

\bibitem{janner2017self}
M.~Janner, J.~Wu, T.~D. Kulkarni, I.~Yildirim, and J.~Tenenbaum,
  ``Self-supervised intrinsic image decomposition,'' in \emph{Proc. Advances in
  neural information processing systems (NIPS)}, 2017, pp. 5936--5946.

\bibitem{liu2017material}
G.~Liu, D.~Ceylan, E.~Yumer, J.~Yang, and J.-M. Lien, ``Material editing using
  a physically based rendering network,'' in \emph{Proc. International
  Conference on Computer Vision (ICCV)}, 2017, pp. 2261--2269.

\bibitem{li2018learning}
Z.~Li, Z.~Xu, R.~Ramamoorthi, K.~Sunkavalli, and M.~Chandraker, ``Learning to
  reconstruct shape and spatially-varying reflectance from a single image,'' in
  \emph{{ACM} Trans. Graph. (Proceedings of {SIGGRAPH} Asia)}, 2018, p. 269.

\bibitem{sengupta2019neural}
S.~Sengupta, J.~Gu, K.~Kim, G.~Liu, D.~W. Jacobs, and J.~Kautz, ``Neural
  inverse rendering of an indoor scene from a single image,'' in \emph{Proc.
  International Conference on Computer Vision (ICCV)}, 2019.

\bibitem{li2020inverse}
Z.~Li, M.~Shafiei, R.~Ramamoorthi, K.~Sunkavalli, and M.~Chandraker, ``Inverse
  rendering for complex indoor scenes: Shape, spatially-varying lighting and
  {SVBRDF} from a single image,'' in \emph{Proc. IEEE Conference on Computer
  Vision and Pattern Recognition (CVPR)}, 2020, pp. 2475--2484.

\bibitem{ronneberger2015u}
O.~Ronneberger, P.~Fischer, and T.~Brox, ``U-net: Convolutional networks for
  biomedical image segmentation,'' in \emph{International Conference on Medical
  image computing and computer-assisted intervention}.\hskip 1em plus 0.5em
  minus 0.4em\relax Springer, 2015, pp. 234--241.

\bibitem{zhao2017pyramid}
H.~Zhao, J.~Shi, X.~Qi, X.~Wang, and J.~Jia, ``Pyramid scene parsing network,''
  in \emph{Proc. IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR)}, 2017, pp. 2881--2890.

\bibitem{golub1973differentiation}
G.~H. Golub and V.~Pereyra, ``The differentiation of pseudo-inverses and
  nonlinear least squares problems whose variables separate,'' \emph{SIAM
  Journal on numerical analysis}, vol.~10, no.~2, pp. 413--432, 1973.

\bibitem{schonberger2016structure}
J.~L. Schonberger and J.-M. Frahm, ``Structure-from-motion revisited,'' in
  \emph{Proc. IEEE Conference on Computer Vision and Pattern Recognition
  (CVPR)}, 2016, pp. 4104--4113.

\bibitem{zhang2016colorful}
R.~Zhang, P.~Isola, and A.~A. Efros, ``Colorful image colorization,'' in
  \emph{Proc. European Conference on Computer Vision (ECCV)}.\hskip 1em plus
  0.5em minus 0.4em\relax Springer, 2016, pp. 649--666.

\bibitem{johnson2016perceptual}
J.~Johnson, A.~Alahi, and L.~Fei-Fei, ``Perceptual losses for real-time style
  transfer and super-resolution,'' in \emph{Proc. European Conference on
  Computer Vision (ECCV)}, 2016, pp. 694--711.

\bibitem{dror2001statistics}
R.~O. Dror, T.~K. Leung, E.~H. Adelson, and A.~S. Willsky, ``Statistics of
  real-world illumination,'' in \emph{Proc. IEEE Conference on Computer Vision
  and Pattern Recognition (CVPR)}, 2001.

\bibitem{hdrlabs}
\BIBentryALTinterwordspacing
H.~Labs, \emph{sIBL Archive}, 2007--2012. [Online]. Available:
  \url{http://www.hdrlabs.com/sibl/archive.html}
\BIBentrySTDinterwordspacing

\bibitem{hdriskies}
\BIBentryALTinterwordspacing
\emph{HDRI-Skies}, 2020. [Online]. Available: \url{https://hdri-skies.com/}
\BIBentrySTDinterwordspacing

\bibitem{kingma2014adam}
D.~P. Kingma and J.~Ba, ``Adam: A method for stochastic optimization,''
  \emph{arXiv preprint arXiv:1412.6980}, 2014.

\bibitem{nestmeyer2017reflectance}
T.~Nestmeyer and P.~V. Gehler, ``Reflectance adaptive filtering improves
  intrinsic image estimation,'' in \emph{Proc. IEEE Conference on Computer
  Vision and Pattern Recognition (CVPR)}, vol.~2, no.~3, 2017, p.~4.

\bibitem{bell14intrinsic}
S.~Bell, K.~Bala, and N.~Snavely, ``Intrinsic images in the wild,'' \emph{{ACM}
  Trans. Graph. (Proceedings of {SIGGRAPH})}, vol.~33, no.~4, 2014.

\bibitem{diode_dataset}
\BIBentryALTinterwordspacing
I.~Vasiljevic, N.~Kolkin, S.~Zhang, R.~Luo, H.~Wang, F.~Z. Dai, A.~F. Daniele,
  M.~Mostajabi, S.~Basart, M.~R. Walter, and G.~Shakhnarovich, ``{DIODE}: {A}
  {D}ense {I}ndoor and {O}utdoor {DE}pth {D}ataset,'' \emph{CoRR}, vol.
  abs/1908.00463, 2019. [Online]. Available:
  \url{http://arxiv.org/abs/1908.00463}
\BIBentrySTDinterwordspacing

\bibitem{yu20relightNet}
Y.~Yu, A.~Meka, M.~Elgharib, H.-P. Seidel, C.~Theobalt, and W.~A.~P. Smith,
  ``Self-supervised outdoor scene relighting,'' in \emph{Proc. European
  Conference on Computer Vision (ECCV)}, 2020.

\bibitem{Silberman:ECCV12}
P.~K. Nathan~Silberman, Derek~Hoiem and R.~Fergus, ``Indoor segmentation and
  support inference from rgbd images,'' in \emph{ECCV}, 2012.

\bibitem{shi2017learning}
J.~Shi, Y.~Dong, H.~Su, and X.~Y. Stella, ``Learning non-lambertian object
  intrinsics across shapenet categories,'' in \emph{Proc. IEEE Conference on
  Computer Vision and Pattern Recognition (CVPR)}, 2017, pp. 5844--5853.

\bibitem{kleffner1992perception}
D.~A. Kleffner and V.~S. Ramachandran, ``On the perception of shape from
  shading,'' \emph{Perception \& Psychophysics}, vol.~52, no.~1, pp. 18--36,
  1992.

\bibitem{mather2017turbine}
G.~Mather and R.~Lee, ``Turbine blade illusion,'' \emph{i-Perception}, vol.~8,
  no.~3, p. 2041669517710031, 2017.

\bibitem{yu2019depth}
Y.~Yu and W.~A.~P. Smith, ``Depth estimation meets inverse rendering for single
  image novel view synthesis,'' in \emph{Proc. of the ACM SIGGRAPH European
  Conference on Visual Media Production (CVMP)}, 2019.

\bibitem{zou2018layoutnet}
C.~Zou, A.~Colburn, Q.~Shan, and D.~Hoiem, ``Layoutnet: Reconstructing the 3d
  room layout from a single rgb image,'' in \emph{Proc. IEEE Conference on
  Computer Vision and Pattern Recognition (CVPR)}, 2018, pp. 2051--2059.

\end{thebibliography}

refs.bib
% ABBREVIATED TITLES

% Journals

 @String{ACMTOG = "{ACM} Trans. Graph." }
 @String{ACMTOGSIGGRAPH = "{ACM} Trans. Graph. (Proceedings of {SIGGRAPH})" }
 @String{ACMTOGSIGASIA = "{ACM} Trans. Graph. (Proceedings of {SIGGRAPH} Asia)" }
 @String{IEEETransPAMI = "{IEEE} Trans. Pattern Anal. Mach. Intell." }
 @String{IJCV = "Int. J. Comput. Vis." }
 @String{CVIU = "Comput. Vis. Image Underst." }
 
% Conferences
 
 @String{CVPR = "Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)"}
 @String{CVPRW = "Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops"}
 @String{ICCV = "Proc. International Conference on Computer Vision (ICCV)"}
 @String{ECCV = "Proc. European Conference on Computer Vision (ECCV)"}
 @String{ThreeDV = "Proc. International Conference on 3D Vision (3DV)"}
 @String{WACV = "Proc. IEEE Winter Conference on Applications of Computer Vision (WACV)"}
 @String{NIPS = "Proc. Advances in neural information processing systems (NIPS)"}

% REFERENCES

@Article{PGZED19,
  author       = "Philip, Julien and Gharbi, Micha{\"e}l and Zhou, Tinghui and Efros, Alexei and Drettakis, George",
  title        = "Multi-view Relighting Using a Geometry-Aware Network",
  journal      = ACMTOGSIGGRAPH,
  number       = "4",
  volume       = "38",
  month        = "July",
  year         = "2019",
  url          = "http://www-sop.inria.fr/reves/Basilic/2019/PGZED19"
}

@inproceedings{yu2019inverserendernet,
    title={{InverseRenderNet}: Learning single image inverse rendering},
    author={Yu, Ye and Smith, William AP},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2019}
}

@article{worthington2005reillumination,
  title={Reillumination-driven shape from shading},
  author={Worthington, Philip L},
  journal=CVIU,
  volume={98},
  number={2},
  pages={325--343},
  year={2005}
}

@inproceedings{dror2001statistics,
  title={Statistics of real-world illumination},
  author={Dror, Ron O and Leung, Thomas K and Adelson, Edward H and Willsky, Alan S},
  booktitle=CVPR,
  year={2001}
}

@inproceedings{shelhamer2015scene,
  title={Scene intrinsics and depth from a single image},
  author={Shelhamer, Evan and Barron, Jonathan T and Darrell, Trevor},
  booktitle=CVPRW,
  pages={37--44},
  year={2015}
}

@article{aittala2016reflectance,
  title={Reflectance modeling by neural texture synthesis},
  author={Aittala, Miika and Aila, Timo and Lehtinen, Jaakko},
  journal=ACMTOG,
  volume={35},
  number={4},
  pages={65},
  year={2016},
  publisher={ACM}
}
@article{li2017modeling,
  title={Modeling surface appearance from a single photograph using self-augmented convolutional neural networks},
  author={Li, Xiao and Dong, Yue and Peers, Pieter and Tong, Xin},
  journal=ACMTOG,
  volume={36},
  number={4},
  pages={45},
  year={2017},
  publisher={ACM}
}

@article{aldrian2013inverse,
  title={Inverse rendering of faces with a 3D morphable model.},
  author={Aldrian, O and Smith, WA},
  journal=IEEETransPAMI,
  volume={35},
  number={5},
  pages={1080--1093},
  year={2013}
}

@inproceedings{Ramamoorthi:2001,
 author = {Ramamoorthi, Ravi and Hanrahan, Pat},
 title = {An Efficient Representation for Irradiance Environment Maps},
 booktitle = {Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques},
 series = {SIGGRAPH '01},
 year = {2001},
 pages = {497--500}
} 

@article{Snavely:08,
  title={Multi-View Stereo for Community Photo Collections},
  author={Michael Goesele and Noah Snavely and Brian Curless and Hugues Hoppe and Steven M. Seitz},
  journal=ICCV,
  year={2007},
  pages={1-8}
}

@ARTICLE{Lowe:04,
    author = {David G. Lowe},
    title = {Distinctive Image Features from Scale-Invariant Keypoints},
    journal = IJCV,
    year = {2004},
    volume = {60},
    pages = {91--110}
}

@article{Furukawa:10,
 author = {Furukawa, Yasutaka and Ponce, Jean},
 title = {Accurate, Dense, and Robust Multiview Stereopsis},
 journal = IEEETransPAMI,
 volume = {32},
 number = {8},
 year = {2010},
 pages = {1362--1376}
} 

@article{Kazhdan:13,
 author = {Kazhdan, Michael and Hoppe, Hugues},
 title = {Screened {P}oisson Surface Reconstruction},
 journal = ACMTOG,
 volume = {32},
 number = {3},
 year = {2013},
 pages = {29:1--29:13}
} 

@inproceedings{Langguth:12,
 author = {Langguth, Fabian},
 title = {Photometric Stereo for Outdoor Webcams},
 booktitle = CVPR,
 year = {2012},
 pages = {262--269}
} 

@INPROCEEDINGS{Haber:09, 
    author={T. Haber and C. Fuchs and P. Bekaer and H. P. Seidel and M. Goesele and H. P. A. Lensch}, 
    booktitle=CVPR, 
    title={Relighting objects from image collections}, 
    year={2009}, 
    pages={627-634}
}

@article{Goldman:09,
  title={Shape and spatially-varying BRDFs from photometric stereo},
  author={Goldman, Dan B and Curless, Brian and Hertzmann, Aaron and Seitz, Steven M},
  journal=IEEETransPAMI,
  volume={32},
  number={6},
  pages={1060--1071},
  year={2010}
}

@INPROCEEDINGS{Alldrin:08, 
author={N. Alldrin and T. Zickler and D. Kriegman}, 
booktitle=CVPR, 
title={Photometric stereo with non-parametric and spatially-varying reflectance}, 
year={2008}
}

@InProceedings{jin:04,
  author = 	{Jin, H.
		and Yezzi, A. J.
		and Soatto, S.},
  title = 	{Region-based segmentation on evolving surfaces with application to 3D shape and radiance estimation},
  booktitle = ECCV,
  year = 	{2004},
  pages = 	{114--125}
}

@inProceedings{MegaDepthLi18,
  title={MegaDepth: Learning Single-View Depth Prediction from Internet Photos},
  author={Zhengqi Li and Noah Snavely},
  booktitle=CVPR,
  year={2018}
}

@inproceedings{schonberger2016structure,
  title={Structure-from-motion revisited},
  author={Schonberger, Johannes L and Frahm, Jan-Michael},
  booktitle=CVPR,
  pages={4104--4113},
  year={2016}
}

@inproceedings{lettry2018darn,
  title={{DARN}: a deep adversarial residual network for intrinsic image decomposition},
  author={Lettry, Louis and Vanhoey, Kenneth and Van Gool, Luc},
  booktitle=WACV,
  pages={1359--1367},
  year={2018}
}

@inproceedings{chen2013simple,
  title={A simple model for intrinsic image decomposition with depth cues},
  author={Chen, Qifeng and Koltun, Vladlen},
  booktitle=ICCV,
  pages={241--248},
  year={2013}
}

@inproceedings{kim2016multi,
  title={Multi-view inverse rendering under arbitrary illumination and albedo},
  author={Kim, Kichang and Torii, Akihiko and Okutomi, Masatoshi},
  booktitle=ECCV,
  pages={750--767},
  year={2016}
}

@inproceedings{jeon2014intrinsic,
  title={Intrinsic image decomposition using structure-texture separation and surface normals},
  author={Jeon, Junho and Cho, Sunghyun and Tong, Xin and Lee, Seungyong},
  booktitle=ECCV,
  pages={218--233},
  year={2014}
}

@inproceedings{eigen2014depth,
  title={Depth map prediction from a single image using a multi-scale deep network},
  author={Eigen, David and Puhrsch, Christian and Fergus, Rob},
  booktitle=NIPS,
  pages={2366--2374},
  year={2014}
}

@inproceedings{eigen2015predicting,
  title={Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture},
  author={Eigen, David and Fergus, Rob},
  booktitle=ICCV,
  pages={2650--2658},
  year={2015}
}

@inproceedings{laina2016deeper,
  title={Deeper depth prediction with fully convolutional residual networks},
  author={Laina, Iro and Rupprecht, Christian and Belagiannis, Vasileios and Tombari, Federico and Navab, Nassir},
  booktitle=ThreeDV,
  pages={239--248},
  year={2016}
}

@inproceedings{newell2016stacked,
  title={Stacked hourglass networks for human pose estimation},
  author={Newell, Alejandro and Yang, Kaiyu and Deng, Jia},
  booktitle=ECCV,
  pages={483--499},
  year={2016}
}

@inproceedings{zhao2017pyramid,
  title={Pyramid scene parsing network},
  author={Zhao, Hengshuang and Shi, Jianping and Qi, Xiaojuan and Wang, Xiaogang and Jia, Jiaya},
  booktitle=CVPR,
  pages={2881--2890},
  year={2017}
}


@inproceedings{tulsiani2017multi,
  title={Multi-view supervision for single-view reconstruction via differentiable ray consistency},
  author={Tulsiani, Shubham and Zhou, Tinghui and Efros, Alexei A and Malik, Jitendra},
  booktitle=CVPR,
  volume={1},
  number={2},
  pages={3},
  year={2017}
}


@inproceedings{liu2015deep,
  title={Deep convolutional neural fields for depth estimation from a single image},
  author={Liu, Fayao and Shen, Chunhua and Lin, Guosheng},
  booktitle=CVPR,
  pages={5162--5170},
  year={2015}
}

@inproceedings{wang2015towards,
  title={Towards unified depth and semantic prediction from a single image},
  author={Wang, Peng and Shen, Xiaohui and Lin, Zhe and Cohen, Scott and Price, Brian and Yuille, Alan L},
  booktitle=CVPR,
  pages={2800--2809},
  year={2015}
}



@inproceedings{xu2017multi,
  title={Multi-scale continuous {CRF}s as sequential deep networks for monocular depth estimation},
  author={Xu, Dan and Ricci, Elisa and Ouyang, Wanli and Wang, Xiaogang and Sebe, Nicu},
  booktitle=CVPR,
  volume={1},
  year={2017}
}


@inproceedings{fu2018deep,
  title={Deep Ordinal Regression Network for Monocular Depth Estimation},
  author={Fu, Huan and Gong, Mingming and Wang, Chaohui and Batmanghelich, Kayhan and Tao, Dacheng},
  booktitle=CVPR,
  pages={2002--2011},
  year={2018}
}

@inproceedings{liu2017material,
  title={Material editing using a physically based rendering network},
  author={Liu, Guilin and Ceylan, Duygu and Yumer, Ersin and Yang, Jimei and Lien, Jyh-Ming},
  booktitle=ICCV,
  pages={2261--2269},
  year={2017}
}

@inproceedings{li2018learning,
  title={Learning to reconstruct shape and spatially-varying reflectance from a single image},
  author={Li, Zhengqin and Xu, Zexiang and Ramamoorthi, Ravi and Sunkavalli, Kalyan and Chandraker, Manmohan},
  booktitle=ACMTOGSIGASIA,
  pages={269},
  year={2018}
}

@inproceedings{ummenhofer2017demon,
  title={Demon: Depth and motion network for learning monocular stereo},
  author={Ummenhofer, Benjamin and Zhou, Huizhong and Uhrig, Jonas and Mayer, Nikolaus and Ilg, Eddy and Dosovitskiy, Alexey and Brox, Thomas},
  booktitle=CVPR,
  year={2017}
}

@inproceedings{kendall2017end,
  title={End-to-end learning of geometry and context for deep stereo regression},
  author={Kendall, Alex and Martirosyan, Hayk and Dasgupta, Saumitro and Henry, Peter and Kennedy, Ryan and Bachrach, Abraham and Bry, Adam},
  booktitle=ICCV,
  pages={66--75},
  year={2017}
}

@inproceedings{garg2016unsupervised,
  title={Unsupervised {CNN} for single view depth estimation: Geometry to the rescue},
  author={Garg, Ravi and BG, Vijay Kumar and Carneiro, Gustavo and Reid, Ian},
  booktitle=ECCV,
  pages={740--756},
  year={2016}
}


@inproceedings{godard2017unsupervised,
  title={Unsupervised monocular depth estimation with left-right consistency},
  author={Godard, Cl{\'e}ment and Mac Aodha, Oisin and Brostow, Gabriel J},
  booktitle=CVPR,
  year={2017}
}

@inproceedings{zhou2017unsupervised,
  title={Unsupervised learning of depth and ego-motion from video},
  author={Zhou, Tinghui and Brown, Matthew and Snavely, Noah and Lowe, David G},
  booktitle=CVPR,
  volume={2},
  number={6},
  pages={7},
  year={2017}
}

@inproceedings{ji2017surfacenet,
  title={{SurfaceNet}: an end-to-end 3d neural network for multiview stereopsis},
  author={Ji, Mengqi and Gall, Juergen and Zheng, Haitian and Liu, Yebin and Fang, Lu},
  booktitle=ICCV,
  pages={2307--2315},
  year={2017}
}

@article{vijayanarasimhan2017sfm,
  title={{SfM-net}: Learning of structure and motion from video},
  author={Vijayanarasimhan, Sudheendra and Ricco, Susanna and Schmid, Cordelia and Sukthankar, Rahul and Fragkiadaki, Katerina},
  journal={arXiv preprint arXiv:1704.07804},
  year={2017}
}

@inproceedings{wang2018learning,
  title={Learning Depth from Monocular Videos using Direct Methods},
  author={Wang, Chaoyang and Buenaposada, Jos{\'e} Miguel and Zhu, Rui and Lucey, Simon},
  booktitle=CVPR,
  pages={2022--2030},
  year={2018}
}

@article{bell14intrinsic,
	author = "Sean Bell and Kavita Bala and Noah Snavely",
	title = "Intrinsic Images in the Wild",
	journal = ACMTOGSIGGRAPH,
	volume = "33",
	number = "4",
	year = "2014"
}

@inproceedings{chen2016single,
  title={Single-image depth perception in the wild},
  author={Chen, Weifeng and Fu, Zhao and Yang, Dawei and Deng, Jia},
  booktitle=NIPS,
  pages={730--738},
  year={2016}
}

@inproceedings{zheng2018t2net,
  title={T2Net: Synthetic-to-Realistic Translation for Solving Single-Image Depth Estimation Tasks},
  author={Zheng, Chuanxia and Cham, Tat-Jen and Cai, Jianfei},
  booktitle=ECCV,
  pages={767--783},
  year={2018}
}

@inproceedings{narihira2015direct,
  title={Direct intrinsics: Learning albedo-shading decomposition by convolutional regression},
  author={Narihira, Takuya and Maire, Michael and Yu, Stella X},
  booktitle=ICCV,
  pages={2992--2992},
  year={2015}
}

@inproceedings{janner2017self,
  title={Self-supervised intrinsic image decomposition},
  author={Janner, Michael and Wu, Jiajun and Kulkarni, Tejas D and Yildirim, Ilker and Tenenbaum, Josh},
  booktitle=NIPS,
  pages={5936--5946},
  year={2017}
}

@article{han2018learning,
  title={Learning an Intrinsic Image Decomposer Using Synthesized RGB-D Dataset},
  author={Han, Guangyun and Xie, Xiaohua and Lai, Jianhuang and Zheng, Wei-Shi},
  journal={IEEE Signal Processing Letters},
  volume={25},
  number={6},
  pages={753--757},
  year={2018},
  publisher={IEEE}
}

@inproceedings{fan2017revisiting,
  title={Revisiting deep intrinsic image decompositions},
  author={Fan, Qingnan and Yang, Jiaolong and Hua, Gang and Chen, Baoquan and Wipf, David},
  booktitle=CVPR,
  pages={8944--8952},
  year={2018}
}

@inProceedings{BigTimeLi18,
  title={Learning Intrinsic Image Decomposition from Watching the World},
  author={Zhengqi Li and Noah Snavely},
  booktitle=CVPR,
  year={2018}
}

@inProceedings{li2018cgintrinsics,
  title={{CGIntrinsics}: Better Intrinsic Image Decomposition through Physically-Based Rendering},
  author={Zhengqi Li and Noah Snavely},
  booktitle=ECCV,
  year={2018}
}

@inproceedings{ma2018single,
  title={Single Image Intrinsic Decomposition without a Single Intrinsic Image},
  author={Ma, Wei-Chiu and Chu, Hang and Zhou, Bolei and Urtasun, Raquel and Torralba, Antonio},
  booktitle=ECCV,
  pages={201--217},
  year={2018}
}

@inproceedings{taniai2018neural,
  title={Neural Inverse Rendering for General Reflectance Photometric Stereo},
  author={Taniai, Tatsunori and Maehara, Takanori},
  booktitle={Proc. International Conference on Machine Learning (ICML)},
  pages={4864--4873},
  year={2018}
}

@inproceedings {e.20181172,
booktitle = {Eurographics Symposium on Rendering - Experimental Ideas \& Implementations},
editor = {Jakob, Wenzel and Hachisuka, Toshiya},
title = {{Deep Hybrid Real and Synthetic Training for Intrinsic Decomposition}},
author = {Bi, Sai and Kalantari, Nima Khademi and Ramamoorthi, Ravi},
year = {2018},
publisher = {The Eurographics Association},
ISSN = {1727-3463},
ISBN = {978-3-03868-068-0},
DOI = {10.2312/sre.20181172}
}


@InProceedings{Baslamisli_2018_CVPR,
author = {Baslamisli, Anil S. and Le, Hoang-An and Gevers, Theo},
title = {{CNN} Based Learning Using Reflection and Retinex Models for Intrinsic Image Decomposition},
booktitle = CVPR,
month = {June},
year = {2018}
}

@inproceedings{tewari2017mofa,
  title={Mofa: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction},
  author={Tewari, Ayush and Zollh{\"o}fer, Michael and Kim, Hyeongwoo and Garrido, Pablo and Bernard, Florian and P{\'e}rez, Patrick and Theobalt, Christian},
  booktitle=ICCV,
  volume={2},
  number={3},
  pages={5},
  year={2017}
}

@inproceedings{kulkarni2015deep,
  title={Deep convolutional inverse graphics network},
  author={Kulkarni, Tejas D and Whitney, William F and Kohli, Pushmeet and Tenenbaum, Josh},
  booktitle=NIPS,
  pages={2539--2547},
  year={2015}
}

@inproceedings{shu2017neural,
  title={Neural face editing with intrinsic image disentangling},
  author={Shu, Zhixin and Yumer, Ersin and Hadap, Sunil and Sunkavalli, Kalyan and Shechtman, Eli and Samaras, Dimitris},
  booktitle=CVPR,
  pages={5444--5453},
  year={2017}
}

@inproceedings{sengupta2017sfsnet,
  title={{SfSNet}: Learning Shape, Reflectance and Illuminance of Faces `in the wild'},
  author={Sengupta, Soumyadip and Kanazawa, Angjoo and Castillo, Carlos D and Jacobs, David W},
  booktitle=CVPR,
  year={2018}
}

@ARTICLE{BarronTPAMI2015, 
Author = {Jonathan T Barron and Jitendra Malik},
Title = {Shape, Illumination, and Reflectance from Shading},
journal=IEEETransPAMI,
year={2015}
}

@inproceedings{grosse2009ground,
  title={Ground truth dataset and baseline evaluations for intrinsic image algorithms},
  author={Grosse, Roger and Johnson, Micah K and Adelson, Edward H and Freeman, William T},
  booktitle=ICCV,
  pages={2335--2342},
  year={2009}
}

@inproceedings{garces2012intrinsic,
  title={Intrinsic images by clustering},
  author={Garces, Elena and Munoz, Adolfo and Lopez-Moreno, Jorge and Gutierrez, Diego},
  booktitle={Computer graphics forum},
  volume={31},
  number={4},
  pages={1415--1424},
  year={2012},
  organization={Wiley Online Library}
}


@inproceedings{shen2011intrinsic,
  title={Intrinsic images decomposition using a local and global sparse representation of reflectance},
  author={Shen, Li and Yeo, Chuohao},
  booktitle=CVPR,
  year={2011}
}

@article{Bi:2015,
 author = {Bi, Sai and Han, Xiaoguang and Yu, Yizhou},
 title = {An L1 Image Transform for Edge-preserving Smoothing and Scene-level Intrinsic Decomposition},
 journal = ACMTOG,
 volume = {34},
 number = {4},
 year = {2015},
 pages = {78:1--78:12}
} 

@inproceedings{zhou2015learning,
  title={Learning data-driven reflectance priors for intrinsic image decomposition},
  author={Zhou, Tinghui and Krahenbuhl, Philipp and Efros, Alexei A},
  booktitle=ICCV,
  pages={3469--3477},
  year={2015}
}


@inproceedings{nestmeyer2017reflectance,
  title={Reflectance adaptive filtering improves intrinsic image estimation},
  author={Nestmeyer, Thomas and Gehler, Peter V},
  booktitle=CVPR,
  volume={2},
  number={3},
  pages={4},
  year={2017}
}

@inproceedings{shi2017learning,
  title={Learning non-lambertian object intrinsics across shapenet categories},
  author={Shi, Jian and Dong, Yue and Su, Hao and Stella, X Yu},
  booktitle=CVPR,
  pages={5844--5853},
  year={2017}
}

@article{Gao:2019:DIR, 
author = {Gao, Duan and Li, Xiao and Dong, Yue and Peers, Pieter and Xu, Kun and Tong, Xin},
title = {Deep Inverse Rendering for High Resolution {SVBRDF} Estimation from an Arbitrary Number of Images},
month = {July},
year = {2019},
journal = ACMTOG,
volume = {37},
number = {4}
}

@article{kanamori2018relighting,
  title={Relighting humans: occlusion-aware inverse rendering for full-body human images},
  author={Kanamori, Yoshihiro and Endo, Yuki},
  journal=ACMTOGSIGASIA,
  volume={37},
  number={6},
  year={2018}
}


@article{nestmeyer2019structural,
  title={Structural Decompositions for End-to-End Relighting},
  author={Nestmeyer, Thomas and Matthews, Iain and Lalonde, Jean-Fran{\c{c}}ois and Lehrmann, Andreas M},
  journal={arXiv preprint arXiv:1906.03355},
  year={2019}
}

@inproceedings{sengupta2019neural,
  title={Neural inverse rendering of an indoor scene from a single image},
  author={Sengupta, Soumyadip and Gu, Jinwei and Kim, Kihwan and Liu, Guilin and Jacobs, David W and Kautz, Jan},
  booktitle=ICCV,
  year={2019}
}

@article{diode_dataset,
  title={{DIODE}: {A} {D}ense {I}ndoor and {O}utdoor {DE}pth {D}ataset},
  author={Igor Vasiljevic and Nick Kolkin and Shanyi Zhang and Ruotian Luo and
  Haochen Wang and Falcon Z. Dai and Andrea F. Daniele and Mohammadreza Mostajabi and
  Steven Basart and Matthew R. Walter and Gregory Shakhnarovich},
  journal={CoRR},
  volume={abs/1908.00463},
  year={2019},
  url={http://arxiv.org/abs/1908.00463}
}

@inproceedings{ronneberger2015u,
  title={U-net: Convolutional networks for biomedical image segmentation},
  author={Ronneberger, Olaf and Fischer, Philipp and Brox, Thomas},
  booktitle={International Conference on Medical image computing and computer-assisted intervention},
  pages={234--241},
  year={2015},
  organization={Springer}
}

@inproceedings{Silberman:ECCV12,
  author    = {Nathan Silberman, Derek Hoiem, Pushmeet Kohli and Rob Fergus},
  title     = {Indoor Segmentation and Support Inference from RGBD Images},
  booktitle = {ECCV},
  year      = {2012}
}

@article{golub1973differentiation,
  title={The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate},
  author={Golub, Gene H and Pereyra, Victor},
  journal={SIAM Journal on numerical analysis},
  volume={10},
  number={2},
  pages={413--432},
  year={1973},
  publisher={SIAM}
}

@inproceedings{monodepth2,
  title     = {Digging into Self-Supervised Monocular Depth Prediction},
  author    = {Cl{\'{e}}ment Godard and
               Oisin {Mac Aodha} and
               Michael Firman and
               Gabriel J. Brostow},
  booktitle = ICCV,
  month = {October},
year = {2019}
}

@article{kleffner1992perception,
  title={On the perception of shape from shading},
  author={Kleffner, Dorothy A and Ramachandran, Vilayanur S},
  journal={Perception \& Psychophysics},
  volume={52},
  number={1},
  pages={18--36},
  year={1992}
}

@article{mather2017turbine,
  title={Turbine blade illusion},
  author={Mather, George and Lee, Rob},
  journal={i-Perception},
  volume={8},
  number={3},
  pages={2041669517710031},
  year={2017},
  publisher={SAGE Publications Sage UK: London, England}
}

@inproceedings{yu2019depth,
  title={Depth estimation meets inverse rendering for single image novel view synthesis},
  author={Yu, Ye and Smith, William A. P.},
  booktitle={Proc. of the ACM SIGGRAPH European Conference on Visual Media Production (CVMP)},
  year={2019}
}

@inproceedings{johnson2016perceptual,
  title={Perceptual losses for real-time style transfer and super-resolution},
  author={Johnson, Justin and Alahi, Alexandre and Fei-Fei, Li},
  booktitle=ECCV,
  pages={694--711},
  year={2016}
}

@article{kingma2014adam,
  title={Adam: A method for stochastic optimization},
  author={Kingma, Diederik P and Ba, Jimmy},
  journal={arXiv preprint arXiv:1412.6980},
  year={2014}
}

@inproceedings{li2020inverse,
  title={Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and {SVBRDF} from a single image},
  author={Li, Zhengqin and Shafiei, Mohammad and Ramamoorthi, Ravi and Sunkavalli, Kalyan and Chandraker, Manmohan},
  booktitle=CVPR,
  pages={2475--2484},
  year={2020}
}

@article{lombardi2015reflectance,
  title={Reflectance and illumination recovery in the wild},
  author={Lombardi, Stephen and Nishino, Ko},
  journal=IEEETransPAMI,
  volume={38},
  number={1},
  pages={129--141},
  year={2015},
  publisher={IEEE}
}

@article{oxholm2015shape,
  title={Shape and reflectance estimation in the wild},
  author={Oxholm, Geoffrey and Nishino, Ko},
  journal=IEEETransPAMI,
  volume={38},
  number={2},
  pages={376--389},
  year={2015},
  publisher={IEEE}
}

@inproceedings{lombardi2016radiometric,
  title={Radiometric scene decomposition: Scene reflectance, illumination, and geometry from rgb-d images},
  author={Lombardi, Stephen and Nishino, Ko},
  booktitle=ThreeDV,
  pages={305--313},
  year={2016},
  organization={IEEE}
}

@inproceedings{romeiro2010blind,
  title={Blind reflectometry},
  author={Romeiro, Fabiano and Zickler, Todd},
  booktitle=ECCV,
  pages={45--58},
  year={2010},
  organization={Springer}
}

@inproceedings{romeiro2008passive,
  title={Passive reflectometry},
  author={Romeiro, Fabiano and Vasilyev, Yuriy and Zickler, Todd},
  booktitle=ECCV,
  pages={859--872},
  year={2008},
  organization={Springer}
}


@inproceedings{zhang2016colorful,
  title={Colorful image colorization},
  author={Zhang, Richard and Isola, Phillip and Efros, Alexei A},
  booktitle=ECCV,
  pages={649--666},
  year={2016},
  organization={Springer}
}

@inproceedings{yu20relightNet,
    title={Self-supervised Outdoor Scene Relighting},
    author={Yu, Ye and Meka, Abhimitra and Elgharib, Mohamed and Seidel, Hans-Peter and Theobalt, Christian and Smith, William A. P.},
    booktitle=ECCV,
    year={2020}
}
    
@manual{hdriskies,
      title  = "HDRI-Skies",
      url    = "https://hdri-skies.com/",
      year   = "2020"
    }
    
    @manual{hdrlabs,
      title  = "sIBL Archive",
      author = "HDR Labs",
      url    = "http://www.hdrlabs.com/sibl/archive.html",
      year   = "2007--2012"
    }


@inproceedings{zou2018layoutnet,
  title={Layoutnet: Reconstructing the 3d room layout from a single rgb image},
  author={Zou, Chuhang and Colburn, Alex and Shan, Qi and Hoiem, Derek},
  booktitle=CVPR,
  pages={2051--2059},
  year={2018}
}

