Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
Xinchen Yan$^1$ \quad Jimei Yang$^2$ \quad Ersin Yumer$^2$ \quad Yijie Guo$^1$ \quad Honglak Lee$^{1
3
\vspace*{-0.1in Understanding the 3D world is a fundamental problem in computer vision. % However, learning a good representation of 3D objects is still an open problem due to the high dimensionality of the data and many factors of variation involved. % In this work, we investigate the task of single-view 3D object reconstruction from a learning agent's perspective. % We formulate the learning process as an interaction between 3D and 2D representations and propose an encoder-decoder network with a novel projection loss defined by the perspective transformation. % More importantly, the projection loss enables the unsupervised learning using 2D observation without explicit 3D supervision. % We demonstrate the ability of the model in generating 3D volume from a single 2D image with three sets of experiments: (1) learning from single-class objects; (2) learning from multi-class objects and (3) testing on novel object classes. % Results show superior performance and better generalization ability for 3D object reconstruction when the projection loss is involved. % \vspace*{-0.1in
arxiv.tex
\documentclass{article}

% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading nips_2016
%
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{nips_2016}

\usepackage[final]{nips_2016}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{nips_2016}

\newcommand{\p}{\mathbf{p}}
\newcommand{\PP}{\mathbf{P}}
\newcommand{\T}{\mathbf{T}}
\newcommand{\Q}{\mathbf{Q}}
\newcommand{\C}{\mathbf{C}}
\newcommand{\U}{\mathbf{U}}
\newcommand{\V}{\mathbf{V}}
\newcommand{\G}{\mathbf{G}}

\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{graphicx}
\usepackage{amsmath} % assumes amsmath package installed
\usepackage{amssymb}  % assumes amsmath package installed
\usepackage{gensymb}
\usepackage{xcolor}
\usepackage{multirow}

\newcommand{\cutsubsectionup}{\vspace*{-0.1in}}
\newcommand{\cutsubsectiondown}{\vspace*{-0.08in}}

\usepackage{siunitx}

\usepackage{tabularx, booktabs}

\definecolor{Red}{rgb}{1., 0, 0}

\newcolumntype{Y}{>{\centering\arraybackslash}X}

\newcommand{\ignore}[1]{}
\newcommand{\TODO}[1]{\textcolor{red}{\textbf{TODO: #1}}}
\newcommand{\jimei}[1]{\textcolor{red}{\textbf{Jimei: #1}}}
\newcommand{\ersin}[1]{\textcolor{blue}{\textbf{Ersin: #1}}}
\newcommand{\cname}[1]{{\fontfamily{cmtt}\selectfont {#1}}}

\title{Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision}


% The \author macro works with any number of authors. There are two
% commands used to separate the names and addresses of multiple
% authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to
% break the lines. Using \AND forces a line break at that point. So,
% if LaTeX puts 3 of 4 authors names on the first line, and the last
% on the second line, try using \AND instead of \And before the third
% author name.

\author{
  Xinchen Yan$^1$ \quad Jimei Yang$^2$ \quad Ersin Yumer$^2$ \quad Yijie Guo$^1$ \quad Honglak Lee$^{1,3}$ \\
  $^1$University of Michigan, Ann Arbor\\
  $^2$Adobe Research \\
  $^3$Google Brain \\  
\texttt{\{xcyan,guoyijie,honglak\}@umich.edu}, \texttt{\{jimyang,yumer\}@adobe.com}, \texttt{honglak@google.com}\\
}

\begin{document}
% \nipsfinalcopy is no longer used

\maketitle
\vspace*{-0.3in}


\begin{abstract}
\vspace*{-0.1in}
Understanding the 3D world is a fundamental problem in computer vision.
%
However, learning a good representation of 3D objects is still an open problem due to the high dimensionality of the data and many factors of variation involved.
%
In this work, we investigate the task of single-view 3D object reconstruction from a learning agent's perspective.
%
We formulate the learning process as an interaction between 3D and 2D representations and propose an encoder-decoder network with a novel projection loss defined by the perspective transformation. 
%
More importantly, the projection loss enables the unsupervised learning using 2D observation without explicit 3D supervision.
%
We demonstrate the ability of the model in generating 3D volume from a single 2D image with three sets of experiments: 
(1) learning from single-class objects; (2) learning from multi-class objects and (3) testing on novel object classes.
%
Results show superior performance and better generalization ability for 3D object reconstruction when the projection loss is involved.
%
\vspace*{-0.1in}
\end{abstract}

\vspace*{-0.1in}
\section{Introduction}
\vspace*{-0.1in}

%
Understanding the 3D world is at the heart of successful computer vision applications in robotics, rendering and modeling~\cite{szeliski2010computer}. It is especially important to solve this problem using the most convenient visual sensory data: 2D images. 
%\TODO{cite more recent papers}
%
In this paper, we propose an end-to-end solution to the challenging problem of predicting the underlying true shape of an object given an arbitrary single image observation of it.
%
This problem definition embodies a fundamental challenge: Imagery observations of 3D shapes are interleaved representations of intrinsic properties of the shape itself (e.g., geometry, material), as well as its extrinsic properties that depend on its interaction with the observer and the environment (e.g., orientation, position, and illumination).
%
Physically principled shape understanding should be able to efficiently disentangle such interleaved factors. 


This observation leads to insight that an end-to-end solution to this problem from the perspective of learning agents (neural networks) should involve the following properties:
%
1) the agent should understand the physical meaning of how a 2D observation is generated from the 3D shape, and 2) the agent should be conscious about the outcome of its interaction with the object; more specifically, by moving around the object, the agent should be able to correspond the observations to the viewpoint change.
%
If such properties are embodied in a learning agent, it will be able to disentangle the shape from the extrinsic factors because these factors are trivial to understand in the 3D world.
%
To enable the agent with these capabilities, we introduce a built-in camera system that can transform the 3D object into 2D images in-network. Additionally, we architect the network such that the latent representation disentangles the shape from view changes.
%
More specifically, our network takes as input an object image and predicts its volumetric 3D shape so that the perspective transformations of predicted shape match well with corresponding 2D observations.


We implement this neural network based on a combination of image encoder, volume decoder and perspective transformer (similar to spatial transformer as introduced by Jaderberg et al.~\cite{jaderberg2015spatial}). 
%
During training, the volumetric 3D shape is gradually learned from single-view input and the feedback of other views through back-propagation. 
%
Thus at test time, the 3D shape can be directly generated from a single image.
%
We conduct experimental evaluations using a subset of 3D models from ShapeNetCore~\cite{chang2015shapenet}.
%
Results from single-class and multi-class training demonstrate excellent performance of our network for volumetric 3D reconstruction.
%
Our main contributions are summarized below.
\vspace*{-0.1in}
\begin{itemize}
    \item We show that neural networks are able to predict 3D shape from single-view without using the ground truth 3D volumetric data for training. This is made possible by introducing a 2D silhouette loss function based on perspective transformations.
    \item We train a single network for multi-class 3D object volumetric reconstruction and show its generalization potential to unseen categories.
    \item Compared to training with full azimuth angles, we demonstrate comparatively similar results when training with partial views.
\end{itemize}
\vspace*{-0.1in}

\section{Related Work}
\vspace*{-0.1in}

\paragraph{Representation learning for 3D objects.}
Recently, advances have been made in learning deep neural networks for 3D objects using large-scale CAD databases~\cite{wu20153d,chang2015shapenet}.
%
Wu et al.~\cite{wu20153d} proposed a deep generative model that extends the convolutional deep belief network \cite{lee2009convolutional} to model volumetric 3D shapes. 
%
Different from \cite{wu20153d} that uses volumetric 3D representation, Su et al.~\cite{su2015multi} proposed a multi-view convolutional network for 3D shape categorization with a view-pooling mechanism.
%
These methods focus more on 3D shape recognition instead of 3D shape reconstruction.
%
Recent work~\cite{tatarchenko2015single,qi2016volumetric,girdhar2016learning,choy20163d} attempt to learn a joint representation for both 2D images and 3D shapes.
%
Tatarchenko et al.~\cite{tatarchenko2015single} developed a convolutional network to synthesize unseen 3D views from a single image and demonstrated the synthesized images can be used them to reconstruct 3D shape. 
%
Qi et al.~\cite{qi2016volumetric} introduced a joint embedding by combining volumetric representation and multi-view representation together to improve 3D shape recognition performance.
%
Girdhar et al.~\cite{girdhar2016learning} proposed a generative model for 3D volumetric data and combined it with a 2D image embedding network for single-view 3D shape generation.
%
Choy et al.~\cite{choy20163d} introduce a 3D recurrent neural network (3D-R2N2) based on long-short term memory (LSTM) to predict the 3D shape of an object from a single view or multiple views. %from single-view or multi-view. 
%
Compared to these single-view methods, our 3D reconstruction network is learned end-to-end and the network can be even trained without ground truth volumes.
%

Concurrent to our work, Renzede et al.~\cite{rezende2016unsupervised} introduced a general framework to learn 3D structures from 2D observations with 3D-2D projection mechanism. 
%
Their 3D-2D projection mechanism either has learnable parameters or adopts non-differentiable component using MCMC,
while our perspective projection network is both differentiable and parameter-free.

\paragraph{Representation learning by transformations.}
Learning from transformed sensory data has gained attention~ \cite{memisevic2007unsupervised,hinton2011transforming,reed2014learning,michalski2014modeling,yang2015weakly,jaderberg2015spatial,yumer2016learning} in recent years.
%
Memisevic and Hinton~\cite{memisevic2007unsupervised} introduced a gated Boltzmann machine that models the transformations between image pairs using multiplicative interaction.
%
Reed et al.~\cite{reed2014learning} showed that a disentangled hidden unit representations of Boltzmann Machines (disBM) could be learned based on the transformations on data manifold.
%
Yang et al.~\cite{yang2015weakly} learned out-of-plane rotation of rendered images to obtain disentangled identity and viewpoint units by curriculum learning.
%
Kulkarni et al.~\cite{KulkarniWKT15} proposed to learn a semantically interpretable latent representation from 3D rendered images using variational auto-encoders \cite{kingma2013auto} 
by including specific transformations in mini-batches.
%
Complimentary to convolutional networks, Jaderberg et al.~\cite{jaderberg2015spatial} introduced a differentiable sampling layer that directly incorporates geometric transformations into representation learning.
%
Concurrent to our work, Wu et al.~\cite{wu2016single} proposed a 3D-2D projection layer that enables the learning of 3D object structures using 2D keypoints as annotation.

\vspace*{-0.1in}
\section{Problem Formulation}
\vspace*{-0.1in}
%
In this section, we develop neural networks for reconstructing 3D objects. 
%
From the perspective of a learning agent (e.g., neural network), a natural way to understand one 3D object $X$ is from its 2D views by transformations. 
%
By moving around the 3D object, the agent should be able to recognize its unique features and eventually build a 3D mental model of it as illustrated in Figure~\ref{fig:intro}(a).
%
Assume that $I^{(k)}$ is the 2D image from the $k$-th viewpoint $\alpha^{(k)}$ by projection $I^{(k)} = P(X;{\alpha^{(k)}})$, or rendering in graphics.
%
An object $X$ in a certain scene is the entanglement of shape, color and texture (its intrinsic properties) and the image $I^{(k)}$ is the further entanglement with viewpoint and illumination (extrinsic parameters).
%
%The general goal of 3D object learning is essentially to disentangle intrinsic properties and extrinsic parameters from a single image.
%The general goal of understanding 3D objects is essentially to disentangle intrinsic properties and extrinsic parameters from a single image.
The general goal of understanding 3D objects can be viewed as disentangling intrinsic properties and extrinsic parameters from a single image.
%
%it is a challenging inverse problem in computer vision and machine learning.
%
%
\begin{figure}[t]
\centering
\includegraphics[width=1\linewidth] {Figure_intro.pdf} 
\vspace{-5mm}
\caption{(a) Understanding 3D object from learning agent's perspective; (b) Single-view 3D volume reconstruction with perspective transformation. (c) Illustration of perspective projection. The minimum and maximum disparity in the camera frame are denoted as $d_{min}$ and $d_{max}$.}
\label{fig:intro}
\end{figure}

%
In this paper, we focus on the 3D shape learning by ignoring the color and texture factors, 
and we further simplify the problem by making the following assumptions: 1) the scene is clean white background; 2) the illumination is constant natural lighting.
%
We use the volumetric representation of 3d shape $\V$ where each voxel $\V_i$ is a binary unit.
%
In other words, the voxel equals to one, i.e., $\V_i = 1$, if the $i$-th voxel sapce is occupied by the shape; otherwise $\V_i = 0$.
%
Assuming the 2D silhouette $S^{(k)}$ is obtained from the $k$-th image $I^{(k)}$, we can specify the 3D-2D projection $S^{(k)} = P(\V;{\alpha^{(k)}})$. 
%
Note that 2D silhouette estimation is typically solved by object segmentation in real-world but it becomes trivial in our case due to the white background.
%
%Therefore, the goal of this paper is learning to predict the volumetric 3D shape $\V$ from an image $I^{(k)}$.

In the following sub-sections, we propose a formulation for learning to predict the volumetric 3D shape $\V$ from an image $I^{(k)}$ with and without the 3D volume supervision.

\vspace*{-0.1in}
\subsection{Learning to Reconstruct Volumetric 3D Shape from Single-View}
\vspace*{-0.1in}
%
We consider single-view volumetric 3D reconstruction as a dense prediction problem and develop a convolutional encoder-decoder network for this learning task denoted by $\hat{\V} = f(I^{(k)})$.
%
The encoder network $h(\cdot)$ learns a \emph{viewpoint-invariant} latent representation $h(I^{(k)})$ which is then used by the decoder $g(\cdot)$ to generate the volume $\hat{\V} = g(h(I^{(k)}))$.
%
In case the ground truth volumetric shapes $\V$ are available, the problem can be easily considered as
learning volumetric 3D shapes with a regular reconstruction objective in 3D space: $\mathcal{L}_{vol}(I^{(k)}) = ||f(I^{(k)}) - \V||^2_2$.
%(see Eq.~\eqref{eqn:volume-loss}):
%\begin{equation}
%\label{eqn:volume-loss}
%\end{equation}
%

In practice, however, \emph{the ground truth volumetric 3D shapes may not be available for training}.
%
For example, the agent observes the 2D silhouette via its built-in camera without accessing the volumetric 3D shape.
%
Inspired by the space carving theory~\cite{kutulakos2000theory}, we propose a silhouette-based volumetric loss function. 
%
In particular, we build on the premise that a 2D silhouette $\hat{S}^{(j)}$ projected from the generated volume $\hat{\V}$ under certain camera viewpoint $\alpha^{(j)}$ should match the ground truth 2D silhouette $S^{(j)}$ from image observations. 
%
In other words, if all the generated silhouettes $\hat{S}^{(j)}$ match well with their corresponding ground truth silhouettes $S^{(j)}$ for all $j$'s, then we hypothesize that the generated volume $\hat{\V}$ should be as good as one instance of \textit{visual hull} equivalent class of the ground truth volume $\V$~\cite{kutulakos2000theory}.
%
Therefore, we formulate the learning objective for the $k$-th image as 
\begin{equation}
\mathcal{L}_{proj}(I^{(k)}) = \sum_{j=1}^{n} \mathcal{L}_{proj}^{(j)}(I^{(k)};S^{(j)},\alpha^{(j)}) = \frac{1}{n}  \sum_{j=1}^{n}||P(f(I^{(k)});{\alpha^{(j)}}) - S^{(j)}||^2_2,
\label{eqn:silhouette-loss}
\end{equation}
%
where $j$ is the index of output 2D silhouettes, $n$ is the number of silhouettes used for each input image and $P(\cdot)$ is the 3D-2D projection function.
%
Note that the above training objective Eq.~\eqref{eqn:silhouette-loss} enables training without using ground-truth volumes. 
%
The network diagram is illustrated in Figure~\ref{fig:intro}(b).
%
A more general learning objective is given by a combination of both objectives:
\begin{equation}
\mathcal{L}_{comb}(I^{(k)}) = \lambda_{proj}\mathcal{L}_{proj}(I^{(k)}) + \lambda_{vol}\mathcal{L}_{vol}(I^{(k)}),
\label{eqn:combined-loss}
\end{equation}
%
where $\lambda_{proj}$ and $\lambda_{vol}$ are constants that control the tradeoff between the two losses. 
%If $\lambda_{proj}$ is set to 0, then the training is driven by 3D shape priors; if $\lambda_{vol}$ is set to 0, then the training is driven by bottom-up 2D observations. 

\vspace*{-0.1in}
\subsection{Perspective Transformer Networks}
\vspace*{-0.1in}
As defined previously, 2D silhouette $S^{(k)}$ is obtained via perspective projection given input 3D volume $\V$ and specific camera viewpoint $\alpha^{(k)}$. 
%
In this work, we implement the perspective projection (see Figure~\ref{fig:intro}(c)) with a 4-by-4 transformation matrix $\mathbf{\Theta}_{4\times4}$, where $\mathbf{K}$ is camera calibration matrix and $(\mathbf{R}, \mathbf{t})$ is extrinsic parameters.
\begin{equation}
\mathbf{\Theta}_{4\times4} = 
\begin{bmatrix}
K & 0\\
0^T & 1\\
\end{bmatrix}
\begin{bmatrix}
R & t\\
0^T & 1\\
\end{bmatrix}
\end{equation}
%
For each point $\p_i^s = (x_i^s, y_i^s, z_i^s, 1)$ in 3D world frame, we compute the corresponding point $\p_i^t = (x_i^t, y_i^t, 1, d_i^t)$ in camera frame (plus disparity $d_i^t$) using the inverse of perspective transformation matrix: $\p_i^s \sim \mathbf{\Theta}^{-1}_{4\times4} \p_i^t$.
%
Similar to the spatial transformer network introduced in~\cite{jaderberg2015spatial}, we implement a novel perspective transformation operator that performs dense sampling from input volume (in 3D world frame) to output volume (in camera frame). To obtain the 2D silhouettes from 3D volume, we propose a simple approach using $\max$ operator that flattens the 3D spatial output across disparity dimension. This operation can be treated as an approximation to the \textit{ray-tracing algorithm}.
%

In the experiment, we assume that transformation matrix is always given as input, parametrized by the viewpoint $\alpha$.
%
Again, the 3D point $(x_i^s, y_i^s, z_i^s)$ in input volume $\V \in \mathbb{R}^{H\times W\times D}$ and corresponding point $(x_i^t, y_i^t, d_i^t)$ in output volume $\U \in \mathbb{R}^{H' \times W' \times D'}$ is linked by perspective transformation matrix $\mathbf{\Theta}_{4\times4}$.
%
Here, $(W,H,D)$ and $(W',H',D')$ are the width, height and depth of input and output volume, respectively. 

We summarize the dense sampling step and channel-wise flattening step as follows.
\begin{equation}
\begin{gathered}
U_i = \sum_{n}^{H} \sum_{m}^{W} \sum_{l}^{D} V_{nml} \max(0,1-|x_i^s-m|) \max(0,1-|y_i^s-n|) \max(0,1-|z_i^s-l|) \\
S_{n'm'} = \max_{l'} U_{n'm'l'}\\
\end{gathered}
\end{equation}
%where $U_i$ is the voxel value corresponding to the $i$-th point $(x_i^t, y_i^t, d_i^t)$.
Here, $U_i$ is the $i$-th voxel value corresponding to the point $(x_i^t, y_i^t, d_i^t)$ (where $i\in \{1,...,W'\times H'\times D'\})$.
Note that we use the $\max$ operator for projection instead of summation along one dimension since the volume is represented as a binary cube where the solid voxels have value 1 and empty voxels have value 0.
%
Intuitively, we have the following two observations: (1) each empty voxel will not contribute to the foreground pixel of $S$ from any viewpoint; (2) each solid voxel can contribute to the foreground pixel of $S$ only if it is visible from a specific viewpoint.
%

%
\vspace*{-0.1in}
\subsection{Training}
\vspace*{-0.1in}
%
As the same volumetric 3D shape is expected to be generated from different images of the object, the encoder network is required to learn a 3D view-invariant latent representation 
\begin{equation}
h(I^{(1)}) = h(I^{(2)}) = \cdots = h(I^{(k)})
\end{equation}
This sub-problem itself is a challenging task in computer vision~\cite{yang2015weakly, KulkarniWKT15}.
%
Thus, we adopt a two-stage training procedure: first, we learn the encoder network for a 3D view-invariant latent representation $h(I)$ and then train the volumetric decoder with perspective transformer networks.
%
As shown in~\cite{yang2015weakly}, a disentangled representation of 2D synthetic images can be learned from consecutive rotations with a recurrent network, we pre-train the encoder of our network using a similar curriculum strategy so that the latent representation only contains 3D view-invariant identity information of the object.
%
Once we obtain an encoder network that recognizes the identity of single-view images, we next learn the volume generator regularized by the perspective transformer networks.
%
To encourage the volume decoder to learn a consistent 3D volume from different viewpoints, we include the projections from neighboring viewpoints in each mini-batch so that the network has relatively sufficient information to reconstruct the 3D shape.

\vspace*{-0.1in}
\section{Experiments}
\vspace*{-0.1in}

\paragraph{ShapeNetCore.} This dataset contains about 51,300 unique 3D models from 55 common object categories \cite{chang2015shapenet}. 
%
Each 3D model is rendered from 24 azimuth angles (with steps of 15$^{\circ}$) with fixed elevation angles (30$^{\circ}$) under the same camera and lighting setup.
%
We then crop and rescale the centering region of each image to $64\times64\times3$ pixels. 
%
For each ground truth 3D shape, we create a volume of $32\times32\times32$ voxels from its canonical orientation ($0^{\circ}$).

\begin{figure}[htbp]
\centering
\includegraphics[width=1\linewidth] {Figure_arch.pdf} 
\vspace{-5mm}
\caption{Illustration of network architecture.}
\vspace{-5mm}
\label{fig:net_arch}
\end{figure}

\vspace*{-0.1in}
\paragraph{Network Architecture.}
As shown in Figure~\ref{fig:net_arch},
our encoder-decoder network has three components: a 2D convolutional encoder, a 3D up-convolutional decoder and a perspective transformer networks.
%
The 2D convolutional encoder consists of 3 convolution layers, followed by 3 fully-connected layers (convolution layers have 64, 128 and 256 channels with fixed filter size of $5\times5$; the three fully-connected layers have 1024, 1024 and 512 neurons, respectively).
%
The 3D convolutional decoder consists of one fully-connected layer, followed by 3 convolution layers (the fully-connected layer have $3\times3\times3\times512$ neurons; convolution layers have 256, 96 and 1 channels with filter size of $4\times4\times4$, $5\times5\times5$ and $6\times6\times6$).
%
For perspective transformer networks, we used perspective transformation to project 3D volume to 2D silhouette where the transformation matrix is parametrized by 16 variables and sampling grid is set to $32\times32\times32$. 
%
We use the same network architecture for all the experiments.

\vspace*{-0.1in}
\paragraph{Implementation Details.}
We used the ADAM \cite{kingma2014adam} solver for stochastic optimization in all the experiments. 
%
During the pre-training stage (for encoder), we used mini-batch of size 32, 32, 8, 4, 3 and 2 for training the RNN-1, RNN-2, RNN-4, RNN-8, RNN-12 and RNN-16 as used in Yang et al.~\cite{yang2015weakly}.
%
We used the learning rate $10^{-4}$ for RNN-1, and $10^{-5}$ for the rest of recurrent neural networks.
%
During the fine-tuning stage (for volume decoder), we used mini-batch of size 6 and learning rate $10^{-4}$. 
%
For each object in a mini-batch, we include projections from all 24 views as supervision.
%
The models including the perspective transformer nets are implemented using Torch~\cite{collobert2011torch7}.
%
To download the code, please refer to the project webpage: \url{http://goo.gl/YEJ2H6}.

\vspace*{-0.1in}
\paragraph{Experimental Design.}
As mentioned in the formulation, there are several variants of the model depending on the hyper-parameters of learning objectives $\lambda_{proj}$ and $\lambda_{vol}$.
In the experimental section, we denote the model trained with projection loss only, volume loss only, and combined loss as \textbf{PTN-Proj} (PR), \textbf{CNN-Vol} (VO), and \textbf{PTN-Comb} (CO), respectively.
% In the experimental section, we denote the model trained with projection loss only as \textbf{PTN-Proj} (PR), volume loss only as \textbf{CNN-Vol} (VO), and combined loss as \textbf{PTN-Comb} (CO).

In the experiments, we address the following questions:
%
(1) Will the model trained with combined loss achieve better single-view 3D reconstruction performance over model trained on volume loss only (\text{PTN-Comb} vs. \text{CNN-Vol})?
(2) What is the performance gap between the models with and without ground-truth volumes (\text{PTN-Comb} vs. \text{PTN-Proj})?
(3) How do the three models generalize to instances from unseen categories which are not present in the training set?
%
To answer the questions, we trained the three models under two experimental settings: single category and multiple categories.
%

\vspace*{-0.1in}
\subsection{Training on a single category}
\label{subsection:single-category}
\vspace*{-0.1in}

\begin{figure}[tbhp]
\centering
\begin{tabular*}{\textwidth}{c}
\setlength{\tabcolsep}{11pt}
\begin{tabular}{ccccccccc}
\tiny Input & \tiny GT (310) &\tiny  GT (130) &\tiny  PR (310) &\tiny  PR (130) &\tiny  CO (310) &\tiny  CO (130) & \tiny VO (310) & \tiny VO (130)  
\end{tabular}\\
\hline
\includegraphics[width=1\textwidth] {singleclass_chair_02_view02.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_02_view07.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_02_view12.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_02_view15.png} \\
\hline
\includegraphics[width=1\textwidth] {singleclass_chair_09_view02.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_09_view07.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_09_view14.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_09_view22.png} \\
\hline
\end{tabular*}
\caption{Single-class results. GT: ground truth, PR: PTN-Proj, CO: PTN-Comb, VO: CNN-Vol (Best viewed in digital version. Zoom in for the 3D shape details). The angles are shown in the parenthesis. Please also see more examples and video animations on the project webpage.}
\vspace*{0.15in}
\label{fig:chair_prediction}
\vspace*{-0.2in}
\end{figure}

\begin{table}[b]
\small
\centering
\caption{
Prediction IU using the models trained on \cname{chair} category. 
%
Below, ``\cname{chair}" corresponds to the setting where each object is observable with full azimuth angles,
while ``\cname{chair-N}" corresponds to the setting where each object is only observable with a narrow range (subset) of azimuth angles.
}
%
\begin{tabular}{l||c|c||c|c}
\hline 
\multirow{2}{*}{Method $/$ Evaluation Set}  & \multicolumn{2}{c||}{\cname{chair}} & \multicolumn{2}{c}{\cname{chair-N} }\tabularnewline
\cline{2-5} 
 & training & test & training & test\tabularnewline
\hline 
PTN-Proj:single (no vol. supervision)  & 0.5712  & 0.5027  & 0.4882  & \textbf{0.4583} \tabularnewline
PTN-Comb:single (vol. supervision)  & \textbf{0.6435}  & \textbf{0.5067}  & \textbf{0.5564}  & 0.4429 \tabularnewline
CNN-Vol:single (vol. supervision)  & 0.6390  & 0.4983  & 0.5518  & 0.4380 \tabularnewline
NN search (vol. supervision) & ---  & 0.3557  & ---  & 0.3073 \tabularnewline
\hline 
\end{tabular}
%\vspace*{0.15in}
\label{tab:table_iou_chair}
%\vspace*{-0.2in}
\end{table}

We select \cname{chair} category as the training set for single category experiment.
%
For model comparisons, we first conduct quantitative evaluations on the generated 3D volumes from the test set single-view images.
%
%
For each instance in the test set, we generate one volume per view image (24 volumes generated in total).
%
Given a pair of ground-truth volume and our generated volume (threshold is 0.5), we computed its intersection-over-union (IU) score and the average IU score is calculated over 24 volumes of all the instances in the test set.
%
In addition, we provide a baseline method based on nearest neighbor (NN) search.
%
Specifically, for each of the test image, we extract VGG feature from \cname{fc6} layer (4096-dim vector)~\cite{simonyan2014very} and retrieve the nearest training example using Euclidean distance in the feature space.
%
The ground-truth 3D volume corresponds to the nearest training example is naturally regarded as the retrieval result.

As shown in Table~\ref{tab:table_iou_chair}, the model trained without volume supervision (projection loss) performs as good as model trained with volume supervision (volume loss) on the \cname{chair} category (testing set).
%
In addition to the comparisons of overall IU, we measured the view-dependent IU for each model.
%
As shown in Figure~\ref{fig:viewdep_iou}, the average prediction error (mean IU) changes as we gradually move from the first view to the last view (15$^{\circ}$ to 360$^{\circ}$). 
%
\begin{figure}[b]
\small
\centering
\vspace*{-0.2in}
\hspace{1cm}
\includegraphics[width=0.9\linewidth] {Figure_view2.pdf} 
\caption{View-dependent IU. 
%
For example, 3D reconstruction from 0$^{\circ}$ is more difficult than from 30$^{\circ}$ due to self-occlusion.
}
\label{fig:viewdep_iou}
\vspace{-0.1in}
\end{figure}
%
For visual comparisons, we provide a side-by-side analysis for each of the three models we trained.
%
As shown in Figure~\ref{fig:chair_prediction}, each row shows an independent comparison. 
%
The first column is the 2D image we used as input of the model.
%
The second and third column show the ground-truth 3D volume (same volume rendered from two views for better visualization purpose).
%
Similarly, we list the model trained with projection loss only (\text{PTN-Proj}), combined loss (\text{PTN-Comb}) and volume loss only (\text{CNN-Vol}) from the fourth column up to the ninth column.
%
The volumes predicted by \text{PTN-Proj} and \text{PTN-Comb} faithfully represent the shape. However, the volumes predicted by \text{CNN-Vol} do not form a solid chair shape in some cases.
%

\textbf{Training with partial views.} We also conduct control experiments where each object is only observable from a narrow range of azimuth angles (e.g., 8 out of 24 views such as 0$^\circ$, 15$^\circ$, $\cdots$, 105$^\circ$).  We include the detailed description in the supplementary materials.
%
As shown in Table~\ref{tab:table_iou_chair} (last two columns), performances of all three models drop a little bit but the conclusion is similar: the proposed network (1) learns better 3D shape with projection regularization and (2) is capable of learning the 3D shape by providing 2D observations only. 

\vspace*{-0.15in}
\subsection{Training on multiple categories}
\label{subsection:multi-category}
\vspace*{-0.1in}


\begin{table}[t]
\small
\centering
\caption{
Prediction IU using the models trained on large-scale datasets.
}
\begin{tabular}{l||c|c|c|c|c|c|c}
\hline
Test Category & \cname{airplane} & \cname{bench} & \cname{dresser} & \cname{car} & \cname{chair} & \cname{display} & \cname{lamp} \\
\hline
PTN-Proj:multi & 0.5556 & 0.4924 & 0.6823 & 0.7123 & 0.4494 & 0.5395 & \textbf{0.4223} \\
PTN-Comb:multi & \textbf{0.5836} & 0.5079 & \textbf{0.7109} & \textbf{0.7381} & \textbf{0.4702} & \textbf{0.5473} & 0.4158 \\
CNN-Vol:multi & 0.5747 & \textbf{0.5142} & 0.6975 & 0.7348 & 0.4451 & 0.5390 & 0.3865 \\
NN search & 0.5564 & 0.4875 & 0.5713 & 0.6519 & 0.3512 & 0.3958 & 0.2905 \\
\hline\hline
Test Category & \cname{loudspeaker} & \cname{rifle} & \cname{sofa} & \cname{table} & \cname{telephone} & \cname{vessel} &  \\
\hline
PTN-Proj:multi & \textbf{0.5868} & 0.5987 & 0.6221 & 0.4938 & 0.7504 & \textbf{0.5507} & \\
PTN-Comb:multi & 0.5675 & \textbf{0.6097} & \textbf{0.6534} & \textbf{0.5146} & \textbf{0.7728} & 0.5399 & \\
CNN-Vol:multi & 0.5478 & 0.6031 & 0.6467 & 0.5136 & 0.7692 & 0.5445 & \\
NN search & 0.4600 & 0.5133 & 0.5314 & 0.3097 & 0.6696 & 0.4078 & \\
\hline
\end{tabular}
\vspace*{0.15in}
\label{tab:table_iou_largescale}
\vspace*{-0.2in}
\end{table}

\begin{figure}[t]
\centering
\begin{tabular*}{\textwidth}{c}
\setlength{\tabcolsep}{11pt}
\begin{tabular}{ccccccccc}
\tiny Input & \tiny GT (310) &\tiny  GT (130) &\tiny  PR (310) &\tiny  PR (130) &\tiny  CO (310) &\tiny  CO (130) & \tiny VO (310) & \tiny VO (130)  
\end{tabular}\\
\hline
\includegraphics[width=1\textwidth] {multiclass_table_03_view20.png} \\
\includegraphics[width=1\textwidth] {multiclass_table_06_view04.png} \\
\hline
\includegraphics[width=1\textwidth] {multiclass_lamp_03_view19.png} \\
\includegraphics[width=1\textwidth] {multiclass_lamp_05_view07.png} \\
\hline
\includegraphics[width=1\textwidth] {multiclass_airplane_02_view04.png} \\ 
\includegraphics[width=1\textwidth] {multiclass_airplane_03_view01.png} \\
\hline
\includegraphics[width=1\textwidth] {multiclass_vessel_02_view19.png} \\
\includegraphics[width=1\textwidth] {multiclass_vessel_03_view04.png} \\
\hline
\end{tabular*}
\caption{Multiclass results. GT: ground truth, PR: PTN-Proj, CO: PTN-Comb, VO: CNN-Vol (Best viewed in digital version. Zoom in for the 3D shape details). The angles are shown in the parenthesis. Please also see more examples and video animations on the project webpage.}
%\vspace*{0.15in}
\label{fig:multiclass_prediction}
\vspace*{-0.2in}
\end{figure}

We conducted multiclass experiment using the same setup in the single-class experiment.
%
For multi-category experiment, the training set includes 13 major categories: \cname{airplane}, \cname{bench}, \cname{dresser}, \cname{car}, \cname{chair}, \cname{display}, \cname{lamp}, \cname{loudspeaker}, \cname{rifle}, \cname{sofa}, \cname{table}, \cname{telephone} and \cname{vessel}.
%
We preserved 20\% of instances from each category as testing data.
%
As shown in Table~\ref{tab:table_iou_largescale}, the quantitative results demonstrate (1) model trained with combined loss is superior to volume loss in most cases and (2) model trained with projection loss perform as good as volume/combined loss.
%
%From the visualization results shown in Figure~\ref{fig:multiclass_prediction}, the volumes generated by all three models look reasonably well.
From the visualization results shown in Figure~\ref{fig:multiclass_prediction}, all three models predict volumes reasonably well.
%
There is only subtle performance difference in object part such as the wing of airplane.


\vspace*{-0.1in}
\subsection{Out-of-Category Tests}
\vspace*{-0.1in}

\begin{table}[t]
\small
\centering
\caption{
Prediction IU in out-of-category tests.
}
\begin{tabular}{l||c|c|c|c|c}
\hline
Method / Test Category & \cname{bed} & \cname{bookshelf} & \cname{cabinet} & \cname{motorbike} & \cname{train} \\
\hline
PTN-Proj:single (no vol. supervision) & \textbf{0.1801} & \textbf{0.1707} & \textbf{0.3937} & \textbf{0.1189} & \textbf{0.1550} \\
PTN-Comb:single (vol. supervision) & 0.1507 & 0.1186 & 0.2626 & 0.0643 & 0.1044 \\
CNN-Vol:single (vol. supervision) & 0.1558 & 0.1183 & 0.2588 & 0.0580 & 0.0956\\
\hline
PTN-Proj:multi (no vol. supervision) & \textbf{0.1944} & \textbf{0.3448} & \textbf{0.6484} & \textbf{0.3216} & 0.3670 \\
PTN-Comb:multi (vol. supervision) & 0.1647 & 0.3195 & 0.5257 & 0.1914 & \textbf{0.3744} \\
CNN-Vol:multi (vol. supervision) & 0.1586 & 0.3037 & 0.4977 & 0.2253 & 0.3740 \\
\hline
\end{tabular}
\vspace*{0.15in}
\label{tab:table_iou_outofcategory}
\vspace*{-0.2in}
\end{table}

Ideally, an intelligent agent should have the ability to generalize the knowledge learned from previously seen categories to unseen categories.
%
To this end, we design out-of-category tests for both models trained on a single category and multiple categories, as described in Section~\ref{subsection:single-category} and Section~\ref{subsection:multi-category}, respectively.
%
We select 5 unseen categories from ShapeNetCore: \cname{bed}, \cname{bookshelf}, \cname{cabinet}, \cname{motorbike} and \cname{train} for out-of-category tests.
%
Here, the two categories \cname{cabinet} and \cname{train} are relatively easier than other categories since there might be instances in the training set with similar shapes (e.g., \cname{dresser}, \cname{vessel}, and \cname{airplane}).
%
But the \cname{bed},\cname{bookshelf} and \cname{motorbike} can be considered as completely novel categories in terms of shape.
%

We summarized the quantitative results in Table~\ref{tab:table_iou_outofcategory}.
%
Surprisingly, the model trained on multiple categories still achieves reasonably good overall IU.
%
As shown in Figure~\ref{fig:novel_prediction}, the proposed projection loss generalizes better than model trained using combined loss or volume loss on \cname{train}, \cname{motorbike} and \cname{cabinet}.
%
The observations from the out-of-category tests suggest that (1) generalization from a single category is very challenging, but training from multiple categories can significantly improve generalization, and (2) the projection regularization can help learning a robust representation for better generalization on unseen categories.

\begin{figure}[t]
\centering
\begin{tabular*}{\textwidth}{c}
\setlength{\tabcolsep}{11pt}
\begin{tabular}{ccccccccc}
\tiny Input & \tiny GT (310) &\tiny  GT (130) &\tiny  PR (310) &\tiny  PR (130) &\tiny  CO (310) &\tiny  CO (310) & \tiny VO (130) & \tiny VO (310)  
\end{tabular}\\
\hline
\includegraphics[width=1\textwidth]{multiclass_train_02_view13.png} \\
\includegraphics[width=1\textwidth]{multiclass_train_06_view02.png} \\
\hline
\includegraphics[width=1\textwidth]{multiclass_motorbike_02_view15.png} \\
\includegraphics[width=1\textwidth]{multiclass_motorbike_04_view01.png} \\
\hline
\includegraphics[width=1\textwidth]{multiclass_cabinet_01_view13.png} \\
\includegraphics[width=1\textwidth]{multiclass_cabinet_03_view15.png} \\
\hline
\end{tabular*}
\caption{Out-of-category results. GT: ground truth, PR: PTN-Proj, CO: PTN-Comb, VO: CNN-Vol (Best viewed in digital version. Zoom in for the 3D shape details). The angles are shown in the parenthesis. Please also see more examples and video animations on the project webpage.}
%\vspace*{0.15in}
\label{fig:novel_prediction}
\vspace*{-0.2in}
\end{figure}

\vspace*{-0.1in}
\section{Conclusions}
\vspace*{-0.1in}

In this paper, we investigate the problem of single-view 3D shape reconstruction from a learning agent's perspective.
%
By formulating the learning procedure as the interaction between 3D shape and 2D observation, we propose to learn an encoder-decoder network which takes advantage of the projection transformation as regularization.
%
Experimental results demonstrate (1) excellent performance of the proposed model in reconstructing the object even without ground-truth 3D volume as supervision and (2) the generalization potential of the proposed model to unseen categories.
%

\vspace*{-0.1in}
\subsection* {Acknowledgments}
\vspace*{-0.1in}
This work was supported in part by NSF CAREER IIS-1453651, ONR N00014-13-1-0762, Sloan Research Fellowship, and a gift from Adobe. We acknowledge NVIDIA for the donation of GPUs. We also thank Yuting Zhang, Scott Reed, Junhyuk Oh, Ruben Villegas, Seunghoon Hong, Wenling Shang, Kibok Lee, Lajanugen Logeswaran, Rui Zhang and Yi Zhang for helpful comments and discussions.

\vspace*{-0.1in}
\begingroup
\setlength{\bibsep}{4pt}
\bibliographystyle{abbrv} % we should use this format for final submission
\small
\bibliography{references}
\endgroup

\appendix

%\vspace*{-0.3in}
\section{Details regarding perspective transformer network}
\vspace*{-0.1in}

As defined in the main text, 2D silhouette $S^{(k)}$ is obtained via perspective transformation given input 3D volume $\V$ and specific camera viewpoint $\alpha^{(k)}$. 
%

\paragraph{Perspective Projection.}
\vspace*{-0.1in}

In this work, we implement the perspective projection (see Figure~\ref{fig:pers_proj}) with a 4-by-4 transformation matrix $\mathbf{\Theta}_{4\times4}$, where $\mathbf{K}$ is camera calibration matrix and $(\mathbf{R}, \mathbf{t})$ is extrinsic parameters.
\begin{equation}
\mathbf{\Theta}_{4\times4} = 
\begin{bmatrix}
K & 0\\
0^T & 1\\
\end{bmatrix}
\begin{bmatrix}
R & t\\
0^T & 1\\
\end{bmatrix}
\end{equation}
%
For each point $\p_i^s = (x_i^s, y_i^s, z_i^s, 1)$ in 3D world frame, we compute the corresponding point $\p_i^t = (x_i^t, y_i^t, 1, d_i^t)$ in camera frame (plus disparity $d_i^t$) using the inverse perspective transformation matrix: $\p_i^s \sim \mathbf{\Theta}^ {-1}_{4\times4} \p_i^t$.
%
\begin{figure}[htbp]
\centering
\includegraphics[width=0.5\linewidth] {Figure_perspect_proj.pdf} 
\caption{Illustration of perspective projection. The minimum and maximum disparity in the camera frame are denoted as $d_{min}$ and $d_{max}$}
\vspace*{0.1in}
\label{fig:pers_proj}
\vspace*{-0.2in}
\end{figure}
Similar to the spatial transformer network introduced in~\cite{jaderberg2015spatial}, we implement
a novel perspective transformation operator that performs dense sampling from input volume (in 3D world frame) to output volume (in camer frame).
In the experiment, we assume that transformation matrix is always given as input, parametrized by the viewpoint $\alpha$.
%
\begin{equation}
\begin{pmatrix}
x_i^s\\
y_i^s\\
z_i^s\\
1\\
\end{pmatrix}
=
\begin{bmatrix}
\theta_{11} & \theta_{12} & \theta_{13} & \theta_{14}\\
\theta_{21} & \theta_{22} & \theta_{23} & \theta_{24}\\
\theta_{31} & \theta_{32} & \theta_{33} & \theta_{34}\\
\theta_{41} & \theta_{42} & \theta_{43} & \theta_{44}\\
\end{bmatrix}
^{-1}
\begin{pmatrix}
\tilde{x_i}^t\\
\tilde{y_i}^t\\
\tilde{z_i}^t\\
1\\
\end{pmatrix}
\label{eqn:projection}
%\vspace*{-0.1in}
\end{equation}
In addition, we obtain normalized coordinates by $x_i^t = \frac{\tilde{x_i}^t}{\tilde{z_i}^t}$, $y_i^t = \frac{\tilde{y_i}^t}{\tilde{z_i}^t}$ and $d_i^t =
\frac{1}{\tilde{z_i}^t}$, where $d_i$ is the disparity.

\paragraph{Differentiable Volume Sampling.}
\vspace*{-0.1in}
To obtain the 2D silhouettes from 3D volume, we propose a simple approach using $\max$ operator that flattens the 3D spatial output across disparity dimension.
%
This operation can be treated as an approximation to the \textit{ray-tracing algorithm}.
%
To make the entire sampling process differentiable, we adopt the similar sampling strategy as proposed in~\cite{jaderberg2015spatial}. That is, each point $(x_i^s, y_i^s, z_i^s)$ defines a spatial location where a sampling kernel $k(\cdot)$ is applied to get the value at a particular voxel in the output volume $U$. 
\begin{equation}
U_i = \sum_{n}^{H} \sum_{m}^{W} \sum_{l}^{D} V_{nml} k(x_i^s-m; \Phi_x) k(y_i^s-n; \Phi_y) k(z_i^s-l; \Phi_z) \hspace{3mm} \forall i \in \{1,...,H'W'D'\}\\
\end{equation}
Here, $\Phi_x$, $\Phi_y$ and $\Phi_z$ are parameters of a generic sampling kernel $k(\cdot)$ which defines the interpolation method. We implement bilinear sampling kernel $k(x) = \max(0, 1-|x|)$ in this work.

Finally, we summarize the dense sampling step and channel-wise flattening step as follows.
\begin{equation}
\begin{gathered}
U_i = \sum_{n}^{H} \sum_{m}^{W} \sum_{l}^{D} V_{nml} \max(0,1-|x_i^s-m|) \max(0,1-|y_i^s-n|) \max(0,1-|z_i^s-l|) \\
S_{n'm'} = \max_{l'} U_{n'm'l'}\\
\end{gathered}
\vspace*{-0.1in}
\end{equation}
Note that we use the $\max$ operator for projection instead of summation along one dimension since the volume is represented as a binary cube where the solid voxels have value 1 and empty voxels have value 0.
%
Intuitively, we have the following two observations: (1) each empty voxel will not contribute to the foreground pixel of $S$ from any viewpoint; (2) each solid voxel can contribute to the foreground pixel of $S$ only if it is visible from specific viewpoint.
%
\vspace*{-0.1in}
\section{Details regarding learning from partial views}
\vspace*{-0.1in}
In our experiments, we have access to 2D projections from the entire 24 azimuth angles for each object in the training set. 
%
A natural but more challenging setting is to learn 3D reconstruction given only partial views for each object.
%
To evaluate the performance gap of using partial views during training,
we train the model in two different ways: 1) using a narrow range of azimuths and 2) using sparse azimuths.
%
For the first one, we constrain the azimuth range of 105$^{\circ}$ (8 out of 24 views). 
%
For the second one, we provide 8 views which form full 360$^{\circ}$ rotation but with a larger stepsize of 45$^{\circ}$.
%

For both tasks, we conduct the training based on the new constraints.
%
More specifically, we pre-train the encoder using the method proposed by Yang et al.~\cite{yang2015weakly} with similar curriculum learning strategy: RNN-1, RNN-2, RNN-4 and finally RNN-7 (since only 8 views are available during training).
%
For fine-tuning step, we limit the number of input views based on the constraint.
%
For evaluation in the test set, we use all the views so that the numbers are comparable with original setting.
%
As shown in Table~\ref{tab:table_iou_chair_supp}, performances of all three models drop a little bit. Overall, the proposed network (1) learns better 3D shape with projection regularization and (2) is capable of learning the 3D shape by providing 2D observations only. 
%
Note that the partial view experiments are conducted on single category only, but we believe the results will be consistent in multiple categories.
%
\begin{table}[thbp]
\small
\centering
\caption{
Prediction IU using the models trained on \cname{chair} category. 
%
Here, \cname{chair} corresponds to the setting where each object is observable with full azimuth angles;
\cname{chair-N} corresponds to the setting where each object is only observable with a narrow range of azimuth angles.
\cname{chair-S} corresponds to the setting where each object is only observable with sparse azimuth angles.
}

\begin{tabular}{l||c|c||c|c||c|c}
\hline
\multirow{2}{*}{Method $/$ Evaluation Set}  & \multicolumn{2}{c||}{\cname{chair}} & \multicolumn{2}{c||}{\cname{chair-N}} & 
\multicolumn{2}{c}{\cname{chair-S}}\tabularnewline
\cline{2-7} 
 & training & test & training & test & training & testing\tabularnewline
\hline
PTN-Proj:single  & 0.5712 & 0.5027 & 0.4882 & \textbf{0.4583} & 0.5201 & \textbf{0.4869} \\
PTN-Comb:single  & \textbf{0.6435} & \textbf{0.5067} & \textbf{0.5564} & 0.4429  & \textbf{0.6037} & 0.4682\\
CNN-Vol:single  & 0.6390 & 0.4983 & 0.5518 & 0.4380 & 0.5712 & 0.4646\\
NN search (vol. supervision) & ---  & 0.3557  & ---  & 0.3073 & --- & 0.3084\tabularnewline
\hline
\end{tabular}
\vspace*{0.15in}
\label{tab:table_iou_chair_supp}
\vspace*{-0.2in}
\end{table}
%

\vspace*{-0.1in}
\section{Additional visualization results on 3D volumetric shape reconstruction}
\vspace*{-0.1in}
As shown in Figure~\ref{fig:chair_prediction_supp}, Figure~\ref{fig:multiclass_prediction_supp_1}, Figure~\ref{fig:multiclass_prediction_supp_2} and Figure~\ref{fig:multiclass_prediction_supp_3},
we provide additional side-by-side analysis for each of the three models we trained.
%
In each figure, each row is an independent comparison. 
%
The first column is the 2D image we used as input of the model.
%
The second and third column show the ground-truth 3D volume (same volume rendered from two views for better visualization purpose).
%
Similarly, we list the model trained with projection loss only (\text{PTN-Proj}), combined loss (\text{PTN-Comb}) and volume loss only (\text{CNN-Vol}) from the fourth column up to the ninth column.
%
%The volumes predicted by \text{PTN-Proj} and \text{PTN-Comb} faithfully represent the shape. However, the volumes predicted by \text{CNN-Vol} do not form a solid chair shape in some cases.
%

\begin{figure}[tbhp]
\centering
\begin{tabular*}{\textwidth}{c}
\setlength{\tabcolsep}{11pt}
\begin{tabular}{ccccccccc}
\tiny Input & \tiny GT (310) &\tiny  GT (130) &\tiny  PR (310) &\tiny  PR (130) &\tiny  CO (310) &\tiny  CO (130) & \tiny VO (310) & \tiny VO (130)  
\end{tabular}\\
\hline
\includegraphics[width=1\textwidth] {singleclass_chair_05_view05.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_05_view12.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_05_view16.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_05_view20.png} \\
\hline
\includegraphics[width=1\textwidth] {singleclass_chair_14_view01.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_14_view08.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_14_view13.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_14_view18.png} \\
\hline
\includegraphics[width=1\textwidth] {singleclass_chair_21_view05.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_21_view10.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_21_view16.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_21_view21.png} \\
\hline
\includegraphics[width=1\textwidth] {singleclass_chair_24_view02.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_24_view07.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_24_view13.png} \\
\includegraphics[width=1\textwidth] {singleclass_chair_24_view22.png} \\
\hline
\end{tabular*}
\caption{Singleclass results. GT: ground truth, PR: PTN-Proj, CO: PTN-Comb, VO: CNN-Vol (Best viewed in digital version. Zoom in for the 3D shape details). The angles are shown in the parenthesis. Please also see more examples and video animations on the project webpage.}
\vspace*{0.15in}
\label{fig:chair_prediction_supp}
\vspace*{-0.2in}
\end{figure}

\begin{figure}[t]
\centering
\begin{tabular*}{\textwidth}{c}
\setlength{\tabcolsep}{11pt}
\begin{tabular}{ccccccccc}
\tiny Input & \tiny GT (310) &\tiny  GT (130) &\tiny  PR (310) &\tiny  PR (130) &\tiny  CO (310) &\tiny  CO (130) & \tiny VO (310) & \tiny VO (130)  
\end{tabular}\\
\hline
\includegraphics[width=1\textwidth] {multiclass_vessel_06_view07.png} \\
\includegraphics[width=1\textwidth] {multiclass_vessel_08_view22.png} \\
\includegraphics[width=1\textwidth] {multiclass_vessel_10_view02.png} \\
\includegraphics[width=1\textwidth] {multiclass_vessel_12_view23.png} \\
\hline
\includegraphics[width=1\textwidth] {multiclass_airplane_05_view12.png} \\ 
\includegraphics[width=1\textwidth] {multiclass_airplane_06_view05.png} \\
\includegraphics[width=1\textwidth] {multiclass_airplane_11_view19.png} \\
\includegraphics[width=1\textwidth] {multiclass_airplane_12_view13.png} \\
\hline
\includegraphics[width=1\textwidth] {multiclass_car_01_view06.png} \\
\includegraphics[width=1\textwidth] {multiclass_car_04_view16.png} \\
\includegraphics[width=1\textwidth] {multiclass_car_11_view06.png} \\
\includegraphics[width=1\textwidth] {multiclass_car_12_view20.png} \\
\hline
\includegraphics[width=1\textwidth] {multiclass_rifle_07_view19.png} \\
\includegraphics[width=1\textwidth] {multiclass_rifle_09_view03.png} \\
\includegraphics[width=1\textwidth] {multiclass_rifle_11_view02.png} \\
\includegraphics[width=1\textwidth] {multiclass_rifle_12_view07.png} \\
\hline
\end{tabular*}
\caption{Multiclass results. GT: ground truth, PR: PTN-Proj, CO: PTN-Comb, VO: CNN-Vol (Best viewed in digital version. Zoom in for the 3D shape details). The angles are shown in the parenthesis. Please also see more examples and video animations on the project webpage.}
%\vspace*{0.15in}
\label{fig:multiclass_prediction_supp_1}
\vspace*{-0.2in}
\end{figure}

\begin{figure}[t]
\centering
\begin{tabular*}{\textwidth}{c}
\setlength{\tabcolsep}{11pt}
\begin{tabular}{ccccccccc}
\tiny Input & \tiny GT (310) &\tiny  GT (130) &\tiny  PR (310) &\tiny  PR (130) &\tiny  CO (310) &\tiny  CO (130) & \tiny VO (310) & \tiny VO (130)  
\end{tabular}\\
\hline
\includegraphics[width=1\textwidth] {multiclass_bench_01_view09.png} \\
\includegraphics[width=1\textwidth] {multiclass_bench_02_view15.png} \\
\includegraphics[width=1\textwidth] {multiclass_bench_05_view06.png} \\
\includegraphics[width=1\textwidth] {multiclass_bench_11_view13.png} \\
\hline
\includegraphics[width=1\textwidth] {multiclass_dresser_03_view02.png} \\
\includegraphics[width=1\textwidth] {multiclass_dresser_05_view17.png} \\
\includegraphics[width=1\textwidth] {multiclass_dresser_09_view11.png} \\
\includegraphics[width=1\textwidth] {multiclass_dresser_10_view04.png} \\
\hline
\includegraphics[width=1\textwidth] {multiclass_table_01_view16.png} \\
\includegraphics[width=1\textwidth] {multiclass_table_04_view12.png} \\
\includegraphics[width=1\textwidth] {multiclass_table_02_view19.png} \\
\includegraphics[width=1\textwidth] {multiclass_table_11_view22.png} \\
\hline
\includegraphics[width=1\textwidth] {multiclass_sofa_01_view07.png} \\
\includegraphics[width=1\textwidth] {multiclass_sofa_02_view03.png} \\
\includegraphics[width=1\textwidth] {multiclass_sofa_04_view22.png} \\
\includegraphics[width=1\textwidth] {multiclass_sofa_10_view05.png} \\
\hline
\end{tabular*}
\caption{Multiclass results. GT: ground truth, PR: PTN-Proj, CO: PTN-Comb, VO: CNN-Vol (Best viewed in digital version. Zoom in for the 3D shape details). The angles are shown in the parenthesis. Please also see more examples and video animations on the project webpage.}
%\vspace*{0.15in}
\label{fig:multiclass_prediction_supp_2}
\vspace*{-0.2in}
\end{figure}

\begin{figure}[ht]
\centering
\begin{tabular*}{\textwidth}{c}
\setlength{\tabcolsep}{11pt}
\begin{tabular}{ccccccccc}
\tiny Input & \tiny GT (310) &\tiny  GT (130) &\tiny  PR (310) &\tiny  PR (130) &\tiny  CO (310) &\tiny  CO (130) & \tiny VO (310) & \tiny VO (130)  
\end{tabular}\\
\hline
\includegraphics[width=1\textwidth] {multiclass_lamp_02_view06.png} \\
\includegraphics[width=1\textwidth] {multiclass_lamp_04_view12.png} \\
\includegraphics[width=1\textwidth] {multiclass_lamp_05_view01.png} \\
\includegraphics[width=1\textwidth] {multiclass_lamp_12_view01.png} \\
\hline
\includegraphics[width=1\textwidth] {multiclass_display_01_view20.png} \\
\includegraphics[width=1\textwidth] {multiclass_display_02_view20.png} \\
\includegraphics[width=1\textwidth] {multiclass_display_04_view09.png} \\
\includegraphics[width=1\textwidth] {multiclass_display_08_view11.png} \\
\hline
\end{tabular*}
\caption{Multiclass results. GT: ground truth, PR: PTN-Proj, CO: PTN-Comb, VO: CNN-Vol (Best viewed in digital version. Zoom in for the 3D shape details). The angles are shown in the parenthesis. Please also see more examples and video animations on the project webpage.}
%\vspace*{0.15in}
\label{fig:multiclass_prediction_supp_3}
\vspace*{-0.2in}
\end{figure}


\end{document}

arxiv.bbl
\begin{thebibliography}{24}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
  \providecommand{\doi}[1]{doi: #1}\else
  \providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi

\bibitem[Chang et~al.(2015)Chang, Funkhouser, Guibas, Hanrahan, Huang, Li,
  Savarese, Savva, Song, Su, et~al.]{chang2015shapenet}
A.~X. Chang, T.~Funkhouser, L.~Guibas, P.~Hanrahan, Q.~Huang, Z.~Li,
  S.~Savarese, M.~Savva, S.~Song, H.~Su, et~al.
\newblock Shapenet: An information-rich 3d model repository.
\newblock \emph{arXiv preprint arXiv:1512.03012}, 2015.

\bibitem[Choy et~al.(2016)Choy, Xu, Gwak, Chen, and Savarese]{choy20163d}
C.~B. Choy, D.~Xu, J.~Gwak, K.~Chen, and S.~Savarese.
\newblock 3d-r2n2: A unified approach for single and multi-view 3d object
  reconstruction.
\newblock In \emph{ECCV}, 2016.

\bibitem[Collobert et~al.(2011)Collobert, Kavukcuoglu, and
  Farabet]{collobert2011torch7}
R.~Collobert, K.~Kavukcuoglu, and C.~Farabet.
\newblock Torch7: A matlab-like environment for machine learning.
\newblock In \emph{BigLearn, NIPS Workshop}, 2011.

\bibitem[Girdhar et~al.(2016)Girdhar, Fouhey, Rodriguez, and
  Gupta]{girdhar2016learning}
R.~Girdhar, D.~F. Fouhey, M.~Rodriguez, and A.~Gupta.
\newblock Learning a predictable and generative vector representation for
  objects.
\newblock In \emph{ECCV}, 2016.

\bibitem[Hinton et~al.(2011)Hinton, Krizhevsky, and
  Wang]{hinton2011transforming}
G.~E. Hinton, A.~Krizhevsky, and S.~D. Wang.
\newblock Transforming auto-encoders.
\newblock In \emph{ICANN}. Springer, 2011.

\bibitem[Jaderberg et~al.(2015)Jaderberg, Simonyan, Zisserman,
  et~al.]{jaderberg2015spatial}
M.~Jaderberg, K.~Simonyan, A.~Zisserman, et~al.
\newblock Spatial transformer networks.
\newblock In \emph{NIPS}, 2015.

\bibitem[Kingma and Ba(2014)]{kingma2014adam}
D.~Kingma and J.~Ba.
\newblock Adam: A method for stochastic optimization.
\newblock \emph{arXiv preprint arXiv:1412.6980}, 2014.

\bibitem[Kingma and Welling(2013)]{kingma2013auto}
D.~P. Kingma and M.~Welling.
\newblock Auto-encoding variational bayes.
\newblock \emph{arXiv preprint arXiv:1312.6114}, 2013.

\bibitem[Kulkarni et~al.(2015)Kulkarni, Whitney, Kohli, and
  Tenenbaum]{KulkarniWKT15}
T.~D. Kulkarni, W.~Whitney, P.~Kohli, and J.~B. Tenenbaum.
\newblock Deep convolutional inverse graphics network.
\newblock In \emph{NIPS}, 2015.

\bibitem[Kutulakos and Seitz(2000)]{kutulakos2000theory}
K.~N. Kutulakos and S.~M. Seitz.
\newblock A theory of shape by space carving.
\newblock \emph{International Journal of Computer Vision}, 38\penalty0
  (3):\penalty0 199--218, 2000.

\bibitem[Lee et~al.(2009)Lee, Grosse, Ranganath, and Ng]{lee2009convolutional}
H.~Lee, R.~Grosse, R.~Ranganath, and A.~Y. Ng.
\newblock Convolutional deep belief networks for scalable unsupervised learning
  of hierarchical representations.
\newblock In \emph{ICML}, 2009.

\bibitem[Memisevic and Hinton(2007)]{memisevic2007unsupervised}
R.~Memisevic and G.~Hinton.
\newblock Unsupervised learning of image transformations.
\newblock In \emph{CVPR}, 2007.

\bibitem[Michalski et~al.(2014)Michalski, Memisevic, and
  Konda]{michalski2014modeling}
V.~Michalski, R.~Memisevic, and K.~Konda.
\newblock Modeling deep temporal dependencies with recurrent grammar cells"".
\newblock In \emph{NIPS}, 2014.

\bibitem[Qi et~al.(2016)Qi, Su, Niessner, Dai, Yan, and
  Guibas]{qi2016volumetric}
C.~R. Qi, H.~Su, M.~Niessner, A.~Dai, M.~Yan, and L.~J. Guibas.
\newblock Volumetric and multi-view cnns for object classification on 3d data.
\newblock In \emph{CVPR}, 2016.

\bibitem[Reed et~al.(2014)Reed, Sohn, Zhang, and Lee]{reed2014learning}
S.~Reed, K.~Sohn, Y.~Zhang, and H.~Lee.
\newblock Learning to disentangle factors of variation with manifold
  interaction.
\newblock In \emph{ICML}, 2014.

\bibitem[Rezende et~al.(2016)Rezende, Eslami, Mohamed, Battaglia, Jaderberg,
  and Heess]{rezende2016unsupervised}
D.~J. Rezende, S.~Eslami, S.~Mohamed, P.~Battaglia, M.~Jaderberg, and N.~Heess.
\newblock Unsupervised learning of 3d structure from images.
\newblock In \emph{NIPS}, 2016.

\bibitem[Simonyan and Zisserman(2014)]{simonyan2014very}
K.~Simonyan and A.~Zisserman.
\newblock Very deep convolutional networks for large-scale image recognition.
\newblock \emph{arXiv preprint arXiv:1409.1556}, 2014.

\bibitem[Su et~al.(2015)Su, Maji, Kalogerakis, and Learned-Miller]{su2015multi}
H.~Su, S.~Maji, E.~Kalogerakis, and E.~Learned-Miller.
\newblock Multi-view convolutional neural networks for 3d shape recognition.
\newblock In \emph{ICCV}, 2015.

\bibitem[Szeliski(2010)]{szeliski2010computer}
R.~Szeliski.
\newblock Computer vision: algorithms and applications.
\newblock 2010.

\bibitem[Tatarchenko et~al.(2015)Tatarchenko, Dosovitskiy, and
  Brox]{tatarchenko2015single}
M.~Tatarchenko, A.~Dosovitskiy, and T.~Brox.
\newblock Single-view to multi-view: Reconstructing unseen views with a
  convolutional network.
\newblock \emph{arXiv preprint arXiv:1511.06702}, 2015.

\bibitem[Wu et~al.(2016)Wu, Xue, Lim, Tian, Tenenbaum, Torralba, and
  Freeman]{wu2016single}
J.~Wu, T.~Xue, J.~J. Lim, Y.~Tian, J.~B. Tenenbaum, A.~Torralba, and W.~T.
  Freeman.
\newblock Single image 3d interpreter network.
\newblock In \emph{ECCV}, 2016.

\bibitem[Wu et~al.(2015)Wu, Song, Khosla, Yu, Zhang, Tang, and Xiao]{wu20153d}
Z.~Wu, S.~Song, A.~Khosla, F.~Yu, L.~Zhang, X.~Tang, and J.~Xiao.
\newblock 3d shapenets: A deep representation for volumetric shapes.
\newblock In \emph{CVPR}, 2015.

\bibitem[Yang et~al.(2015)Yang, Reed, Yang, and Lee]{yang2015weakly}
J.~Yang, S.~E. Reed, M.-H. Yang, and H.~Lee.
\newblock Weakly-supervised disentangling with recurrent transformations for 3d
  view synthesis.
\newblock In \emph{NIPS}, 2015.

\bibitem[Yumer and Mitra(2016)]{yumer2016learning}
E.~Yumer and N.~J. Mitra.
\newblock Learning semantic deformation flows with 3d convolutional networks.
\newblock In \emph{ECCV}, 2016.

\end{thebibliography}

references.bbl
\begin{thebibliography}{10}

\bibitem{chang2015shapenet}
A.~X. Chang, T.~Funkhouser, L.~Guibas, P.~Hanrahan, Q.~Huang, Z.~Li,
  S.~Savarese, M.~Savva, S.~Song, H.~Su, et~al.
\newblock Shapenet: An information-rich 3d model repository.
\newblock {\em arXiv preprint arXiv:1512.03012}, 2015.

\bibitem{choy20163d}
C.~B. Choy, D.~Xu, J.~Gwak, K.~Chen, and S.~Savarese.
\newblock 3d-r2n2: A unified approach for single and multi-view 3d object
  reconstruction.
\newblock In {\em ECCV}, 2016.

\bibitem{collobert2011torch7}
R.~Collobert, K.~Kavukcuoglu, and C.~Farabet.
\newblock Torch7: A matlab-like environment for machine learning.
\newblock In {\em BigLearn, NIPS Workshop}, 2011.

\bibitem{girdhar2016learning}
R.~Girdhar, D.~F. Fouhey, M.~Rodriguez, and A.~Gupta.
\newblock Learning a predictable and generative vector representation for
  objects.
\newblock In {\em ECCV}, 2016.

\bibitem{hinton2011transforming}
G.~E. Hinton, A.~Krizhevsky, and S.~D. Wang.
\newblock Transforming auto-encoders.
\newblock In {\em ICANN}. Springer, 2011.

\bibitem{jaderberg2015spatial}
M.~Jaderberg, K.~Simonyan, A.~Zisserman, et~al.
\newblock Spatial transformer networks.
\newblock In {\em NIPS}, 2015.

\bibitem{kingma2014adam}
D.~Kingma and J.~Ba.
\newblock Adam: A method for stochastic optimization.
\newblock {\em arXiv preprint arXiv:1412.6980}, 2014.

\bibitem{kingma2013auto}
D.~P. Kingma and M.~Welling.
\newblock Auto-encoding variational bayes.
\newblock {\em arXiv preprint arXiv:1312.6114}, 2013.

\bibitem{KulkarniWKT15}
T.~D. Kulkarni, W.~Whitney, P.~Kohli, and J.~B. Tenenbaum.
\newblock Deep convolutional inverse graphics network.
\newblock In {\em NIPS}, 2015.

\bibitem{kutulakos2000theory}
K.~N. Kutulakos and S.~M. Seitz.
\newblock A theory of shape by space carving.
\newblock {\em International Journal of Computer Vision}, 38(3):199--218, 2000.

\bibitem{lee2009convolutional}
H.~Lee, R.~Grosse, R.~Ranganath, and A.~Y. Ng.
\newblock Convolutional deep belief networks for scalable unsupervised learning
  of hierarchical representations.
\newblock In {\em ICML}, 2009.

\bibitem{memisevic2007unsupervised}
R.~Memisevic and G.~Hinton.
\newblock Unsupervised learning of image transformations.
\newblock In {\em CVPR}, 2007.

\bibitem{michalski2014modeling}
V.~Michalski, R.~Memisevic, and K.~Konda.
\newblock Modeling deep temporal dependencies with recurrent grammar cells"".
\newblock In {\em NIPS}, 2014.

\bibitem{qi2016volumetric}
C.~R. Qi, H.~Su, M.~Niessner, A.~Dai, M.~Yan, and L.~J. Guibas.
\newblock Volumetric and multi-view cnns for object classification on 3d data.
\newblock In {\em CVPR}, 2016.

\bibitem{reed2014learning}
S.~Reed, K.~Sohn, Y.~Zhang, and H.~Lee.
\newblock Learning to disentangle factors of variation with manifold
  interaction.
\newblock In {\em ICML}, 2014.

\bibitem{rezende2016unsupervised}
D.~J. Rezende, S.~Eslami, S.~Mohamed, P.~Battaglia, M.~Jaderberg, and N.~Heess.
\newblock Unsupervised learning of 3d structure from images.
\newblock In {\em NIPS}, 2016.

\bibitem{simonyan2014very}
K.~Simonyan and A.~Zisserman.
\newblock Very deep convolutional networks for large-scale image recognition.
\newblock {\em arXiv preprint arXiv:1409.1556}, 2014.

\bibitem{su2015multi}
H.~Su, S.~Maji, E.~Kalogerakis, and E.~Learned-Miller.
\newblock Multi-view convolutional neural networks for 3d shape recognition.
\newblock In {\em ICCV}, 2015.

\bibitem{szeliski2010computer}
R.~Szeliski.
\newblock Computer vision: algorithms and applications.
\newblock 2010.

\bibitem{tatarchenko2015single}
M.~Tatarchenko, A.~Dosovitskiy, and T.~Brox.
\newblock Single-view to multi-view: Reconstructing unseen views with a
  convolutional network.
\newblock {\em arXiv preprint arXiv:1511.06702}, 2015.

\bibitem{wu2016single}
J.~Wu, T.~Xue, J.~J. Lim, Y.~Tian, J.~B. Tenenbaum, A.~Torralba, and W.~T.
  Freeman.
\newblock Single image 3d interpreter network.
\newblock In {\em ECCV}, 2016.

\bibitem{wu20153d}
Z.~Wu, S.~Song, A.~Khosla, F.~Yu, L.~Zhang, X.~Tang, and J.~Xiao.
\newblock 3d shapenets: A deep representation for volumetric shapes.
\newblock In {\em CVPR}, 2015.

\bibitem{yang2015weakly}
J.~Yang, S.~E. Reed, M.-H. Yang, and H.~Lee.
\newblock Weakly-supervised disentangling with recurrent transformations for 3d
  view synthesis.
\newblock In {\em NIPS}, 2015.

\bibitem{yumer2016learning}
E.~Yumer and N.~J. Mitra.
\newblock Learning semantic deformation flows with 3d convolutional networks.
\newblock In {\em ECCV}, 2016.

\end{thebibliography}
