

\subsection{Instance-based pose recovery: gas canister}

This section considers the recovery of pose for a specific object instance. This case fits well with many robotics applications where the objects in the environment are known. Moreover, it allows us to establish the accuracy of our approach in a relatively simple setting before dealing with the more challenging object class scenario.

We collected a dataset of 175 RGB-D images of a textureless gas canister. 
The depth data was only used to generate the groundtruth. More specifically, a complete 3D model of the gas canister was reconstructed using KinectFusion \cite{newcombe2011} and the groundtruth object pose for each image was calculated by ICP with careful manual initialization. Then, 10 keypoints were manually defined on the 3D model and projected to the images yielding groundtruth keypoint locations in 2D for training the convnet. 

A random 85\%/15\% split was used for the training/test data. A stacked hourglass network with two hourglass modules was trained. The output heatmaps for the testing images are visualized in the second column of Fig. \ref{fig:gascan}.  As can be seen, the hourglass network is able to locate the keypoints reliably in the presence of viewpoint variety and occlusions. The non-visible keypoints are also well localized due to the network's ability to take global context into account. The estimated object poses are shown in the last two columns of Fig. \ref{fig:gascan}. The projected 3D models align accurately with the image; the full-perspective solution is more precise than the weak-perspective one. It is worth noting that only 150 images were used to train the network from scratch. Overfitting might be an issue with such a small training set, but the empirical results suggest that the hourglass model captures the object appearance very well in this single instance case. More challenging examples with large intra-class variability are considered in Section \ref{sec:pascal}. 

The 6-DoF pose was estimated with the known 3D model and camera intrinsic parameters using the optimization in Section \ref{sec:optimization}. The following geodesic distance was used to measure the rotation error between a pose estimate, $R_1$, and the groundtruth, $R_2$: 
\begin{align}
\Delta(R_1,R_2) = \frac{\|\log(R_1^TR_2)\|_{F}}{\sqrt{2}}.\label{eq:geodesic}
\end{align}
As a simple baseline, the following greedy approach was implemented: the maximum response locations in the heatmaps were selected as 2D keypoint locations and the standard PnP problem was solved by the EPnP algorithm~\cite{lepetit2009} to estimate the object pose. The results are presented in Table~\ref{tab:gas}. While the weak-perspective solutions (Proposed WP) are on average worse than EPnP due to the inaccurate camera model, the full-perspective solutions (Proposed FP) are much more precise than those of EPnP.  The remarkably small pose errors returned by the proposed approach based on a single RGB image are in the range suitable for a general grasping system. In the supplemental video\footnote{https://www.seas.upenn.edu/\%7Epavlakos/projects/object3d}, we demonstrate the efficacy of our approach in this situation by way of a robot grasping experiment. 

\begin{table}
\caption{Pose estimation errors on the gas canister dataset.}
\centering
\renewcommand{\arraystretch}{1.2}
\begin{tabular}{c*{15}{c}}
\hline
 \multirow{2}{*}{Approach}& \multicolumn{2}{c}{Rotation (degree)} & \multicolumn{2}{c}{Translation (mm)} \\
& Mean & Median & Mean & Median \\
\hline
Proposed WP & 7.99 & 7.61 & N/A & N{}/A \\
Proposed FP & 3.57 & 3.11 & 12.05 & 8.82 \\
EPnP \cite{lepetit2009} & 7.17 & 5.21 & 43.45 & 21.51 \\
\hline
\end{tabular}
\vspace{0.25em}
\label{tab:gas}
\end{table}{}

\begin{figure*}{}
  \centering
  \includegraphics[width=0.8\linewidth]{figures/gascan/valid_16.pdf}\\
  \includegraphics[width=0.8\linewidth]{figures/gascan/valid_12.pdf}\\
  \includegraphics[width=0.8\linewidth]{figures/gascan/valid_24.pdf}\\
  \caption{Qualitative results on the gas canister dataset. From left-to-right: RGB images with bounding boxes provided by Faster R-CNN~\cite{ren2015faster}, heatmaps from the convnet, projections of the 3D model with estimated poses using the weak-perspective model and full-perspective model, respectively. Note the better alignment near the handle with the full-perspective model.}\label{fig:gascan}
\end{figure*}

\subsection{Class-based pose recovery: PASCAL3D+}\label{sec:pascal}

Moving to a more challenging scenario, we demonstrate the full strength of our approach using the large-scale PASCAL3D+ dataset \cite{xiang2014}. The stacked hourglass network was trained from scratch with the training set of PASCAL3D+. Instead of training separate models for different object classes, a single network was trained to output heatmap predictions for all of the 124 keypoints from all classes. Using a single network for all keypoints allows us to share features across the available classes and significantly decreases the number of parameters needed for the network. At test time, given the class of the test object, the heatmaps corresponding to the keypoints belonging to this class were extracted. For pose optimization, two cases were tested: (i) the CAD model for the test image was known; and (ii) the CAD model was unknown and the pose was estimated with a deformable model whose basis was learned by PCA on all CAD models for each class in the dataset. Two principal components were used ($k=2$) for each class, which was sufficient to explain greater than $95\%$ of the shape variation. The 3D model was fit to the 2D keypoints with a weak-perspective model as the camera intrinsic parameters were not available. 

\vspace{5pt}\noindent{\bf Semantic correspondences } A crucial component of our approach is the powerful learning procedure that is particularly successful at establishing correspondences across the semantically related keypoints of each class. To demonstrate this network property,  in Fig.~\ref{fig:semantic} we present a subset of the keypoints for each class along with the localizations of these keypoints in a randomly selected set of images among the ones with the top 50 responses. It is interesting to note that despite the large appearance differences due to extreme viewpoint and intra-class variability, the predictions are very consistent and preserve the semantic relation across various class instances.

\begin{figure*}
  \centering
  \includegraphics[width=\linewidth]{figures/pascal3d/valid_semKeyp_9.pdf}\\
  \includegraphics[width=\linewidth]{figures/pascal3d/valid_semKeyp_42.pdf}\\
  \includegraphics[width=\linewidth]{figures/pascal3d/valid_semKeyp_50.pdf}\\
  \includegraphics[width=\linewidth]{figures/pascal3d/valid_semKeyp_61.pdf}\\
  \caption{Localization results for diverse keypoint categories. We visualize eight images selected randomly from the top 50 responses for each keypoint. The keypoint localization network is particularly successful at establishing semantic correspondences across the instances of a class, despite the significant intra-class variation and wide ranging camera viewpoints.}\label{fig:semantic}
\end{figure*}

\vspace{5pt}\noindent{\bf Pose estimation } The quantitative evaluation for pose estimation on PASCAL3D+ is presented in Table~\ref{tab:pascal3d}. Only the errors for rotations are reported as the 3D translation cannot be determined in the weak perspective case and the ground truth is not available as well. The rotational error is calculated using the geodesic distance, (\ref{eq:geodesic}). The proposed method shows improvement across most categories with respect to the state-of-the-art. The best results are achieved in the case where the fine subclass for the object is known and there exists an accurate CAD model correspondence. The proposed method with uniform weights for all keypoints is also compared as a baseline, which is apparently worse than considering the confidences during model fitting. A subset of results of the proposed method are visualized in Fig.~\ref{fig:models3d}.

\begin{figure*}
  \centering
  \includegraphics[width=0.49\linewidth]{figures/pascal3d/aeroplane_valid_1243_model.pdf}
  \includegraphics[width=0.49\linewidth]{figures/pascal3d/aeroplane_valid_1299_model.pdf}
  \includegraphics[width=0.49\linewidth]{figures/pascal3d/bicycle_valid_2475_model.pdf}
  \includegraphics[width=0.49\linewidth]{figures/pascal3d/bottle_valid_5534_model.pdf}
  \includegraphics[width=0.49\linewidth]{figures/pascal3d/bus_valid_6295_model.pdf}
  \includegraphics[width=0.49\linewidth]{figures/pascal3d/car_valid_9982_model.pdf}
  \includegraphics[width=0.49\linewidth]{figures/pascal3d/car_valid_10093_model.pdf}
  \includegraphics[width=0.49\linewidth]{figures/pascal3d/chair_valid_11274_model.pdf}
  \includegraphics[width=0.49\linewidth]{figures/pascal3d/chair_valid_11447_model.pdf} 
  \includegraphics[width=0.49\linewidth]{figures/pascal3d/sofa_valid_15763_model.pdf} 
   \includegraphics[width=0.49\linewidth]{figures/pascal3d/train_valid_16909_model.pdf}
   \includegraphics[width=0.49\linewidth]{figures/pascal3d/train_valid_16710_model.pdf}
  \caption{Example results of our approach on PASCAL3D+. For each example from left-to-right: 
the RGB image of the object, heatmap responses for the keypoints of the specific class, the CAD model projected to 2D after pose estimation, and the CAD model visualized in 3D.
}\label{fig:models3d}
\end{figure*}

\begin{table*}[t]
\caption{Viewpoint Estimation Median Error (degrees) on PASCAL3D+.}
\begin{center}
\begin{tabular}{c|cccccccccc}
\hline
   Approach    &     aero &       bike &  	 bottle &           bus &           car &         chair &	sofa &         train &         TV monitor  &	boat        \\
\hline
Tulsiani and Malik~\cite{tulsiani2015vk}	&         13.8 &         17.7 &     	12.9 &          5.8 &          9.1 &         14.8 &         15.2 &          8.7 &         {\bf 15.4} &    	{\bf 21.3}  \\
ours - PCA basis  &	11.2 &         15.2 &          13.1 &          4.7 &          6.9 &         12.7 &        21.7 &          9.1 &        38.5 &         37.9   \\
ours - CAD basis 	&   {\bf 8.0} &         {\bf 13.4} &        {\bf 11.7} &          {\bf 2.0} &          {\bf 5.5} &         {\bf 10.4} &          {\bf 9.6} &          {\bf 8.3} &          32.9 &         40.7 \\
ours - uniform weights 	&	16.3 &         17.8 &          14.1 &          11.7 &          30.7 &         17.6 &        32.4 &          20.8 &        25.0 &         72.0   \\
\hline
\end{tabular}
\end{center}
\label{tab:pascal3d}
\end{table*}

\vspace{5pt}\noindent{\bf Failure cases } In Table~\ref{tab:pascal3d} we observed higher errors than the state-of-the-art for two classes, namely boat and TV monitor. For most images of TV monitor, there are only four coplanar keypoints. This makes pose estimation an ill-posed problem for the weak perspective case. Figure \ref{fig:failure} illustrates some failure cases because of this ambiguity. For boat we observed that in many cases the objects are very small and there are insufficient cues to discriminate between the front and the back, which makes the keypoint localization extremely hard. In these extreme cases, holistic and discrete viewpoint prediction might be more robust, which could in practice provide a prior to regularize our continuous prediction. We exclude the results for two classes from PASCAL3D+, namely dining table and motorbike, as we observed some inconsistency of the left-right definition in groundtruth annotations, which lead to erroneous training data. Since other approaches (e.g., \cite{tulsiani2015vk}) rely on discrete viewpoint annotations only, this issue is not reported.

\begin{figure}
  \centering
   \includegraphics[width=\linewidth]{figures/pascal3d/tvmonitor_valid_17604_model.pdf}
   \includegraphics[width=\linewidth]{figures/pascal3d/tvmonitor_valid_17633_model.pdf}\\  
  \caption{Failure cases for the TV monitor class of PASCAL3D+. Although the visible keypoints are localized successfully the pose optimization fails because the 4 points are coplanar and the problem becomes ill-posed.}\label{fig:failure}
\vspace{-1em}
\end{figure}

\subsection{Processing time}

On a desktop with an Intel i7 3.4GHz CPU, 8G RAM and a GeForce GTX Titan X 6GB GPU, our pipeline needs around 0.2 seconds for the keypoint localization step and less than 0.1 seconds for the shape fitting step, for a total running time under 0.3 seconds. This makes our approach particularly suitable for applications where near real-time is desired. Moreover, further improvements in the running time are anticipated due to improvements in hardware, particularly with GPUs.
\subsubsection{Weak perspective model}

If the camera intrinsic parameters are unknown, the weak perspective camera model is adopted, which is usually a good approximation to the full perspective case when the camera is relatively far away from the object. In this case, the reprojection error is written as 
\begin{align}
    \xi(\theta) = {\boldsymbol{W}} - s \bar{{\boldsymbol{R}}}\left({\boldsymbol{B}}_0 + \sum_{i=1}^{k} c_i{\boldsymbol{B}}_i\right) - \bar{{\boldsymbol{T}}}{\boldsymbol{1}}^\top,
\end{align}
where $s$ is a scalar, $\bar{{\boldsymbol{R}}}\in\RR{2}{3}$ and $\bar{{\boldsymbol{T}}}\in\R{2}$ denote the first two rows of ${\boldsymbol{R}}$ and ${\boldsymbol{T}}$, respectively, and $\theta=\{s,{\boldsymbol{c}},\bar{{\boldsymbol{R}}},\bar{{\boldsymbol{T}}}\}$.

The problem in \refEq{eq:cost} is continuous and in principal can be locally solved by any gradient-based method. We solve it with a block coordinate descent scheme because of its fast convergence and the simplicity in implementation. We alternately update each of the variables while fixing the others. The updates of $s$, ${\boldsymbol{c}}$ and $\bar{{\boldsymbol{T}}}$ are simply solved using closed-form least squares solutions. The update of $\bar{{\boldsymbol{R}}}$ should consider the $SO(3)$ constraint. Here, the Manopt toolbox \cite{boumal2014manopt} is used to optimize $\bar{{\boldsymbol{R}}}$ over the Stiefel manifold. As the problem in \refEq{eq:cost} is non-convex, we further adopt a convex relaxation approach \cite{zhou20153d} to initialize the optimization. More specifically, we only estimate the pose parameters while fixing the 3D model as the mean shape in the initialization stage. By setting ${\boldsymbol{c}}={\boldsymbol{0}}$ and replacing the orthogonality constraint on $\bar{{\boldsymbol{R}}}$ by the spectral norm regularizer, the problem in \refEq{eq:cost} can be converted to a convex program and solved with global optimality \cite{zhou20153d}.

\subsubsection{Full perspective model}

If the camera intrinsic parameters are known, the full perspective camera model is used, and the residuals are defined as
\begin{align}
\xi(\theta) = \tilde{{\boldsymbol{W}}}{\boldsymbol{Z}}- {{\boldsymbol{R}}}\left({\boldsymbol{B}}_0 + \sum_{i=1}^{k} c_i{\boldsymbol{B}}_i\right) - {{\boldsymbol{T}}}{\boldsymbol{1}}^\top,
\end{align}
where $\tilde{{\boldsymbol{W}}}\in\RR{3}{p}$ represents the normalized homogeneous coordinates of the 2D keypoints and ${\boldsymbol{Z}}$ is a diagonal matrix: 
\begin{align}
{\boldsymbol{Z}} &= \begin{bmatrix}
            z_1 & 0 & \cdots & 0\\
            0 & z_2 & \cdots & 0\\
            \vdots & \vdots & \ddots & \vdots\\
            0 & 0 & \cdots & z_p\\
            \end{bmatrix},
\end{align}
where $z_i$ is the depth for the $i$th keypoint in 3D. Intuitively, the distances from the 3D points to the rays crossing the corresponding 2D points are minimized. In this case, the unknown parameter set $\theta$ is given by $\{{\boldsymbol{Z}},{\boldsymbol{c}},{\boldsymbol{R}},{\boldsymbol{T}}\}$.

The optimization here is similar to the alternating scheme in the weak perspective case. The update of ${\boldsymbol{Z}}$ also admits a closed-form solution and the update of ${\boldsymbol{R}}$ can be analytically solved by the orthogonal Procrustes analysis. To avoid local minima, the optimization is initialized by the weak perspective solution.  



\subsection{Keypoint localization}
\input{keypoints}

\subsection{Pose optimization}\label{sec:optimization}
\input{fitting}

\section{Introduction}

\input{intro}

\section{Related Work}
\input{related_work}
\input{contributions}

\section{Technical Approach}
\input{methods}

\section{Experiments}
\input{evaluation}

\section{Summary}
\input{summary}



\bibliographystyle{IEEEtran}

