---
abstract: 'We propose a novel approach to 3D human pose estimation from a single depth map. Recently, convolutional neural network (CNN) has become a powerful paradigm in computer vision. Many of computer vision tasks have benefited from CNNs, however, the conventional approach to directly regress 3D body joint locations from an image does not yield a noticeably improved performance. In contrast, we formulate the problem as estimating per-voxel likelihood of key body joints from a 3D occupancy grid. We argue that learning a mapping from volumetric input to volumetric output with 3D convolution consistently improves the accuracy when compared to learning a regression from depth map to 3D joint coordinates. We propose a two-stage approach to reduce the computational overhead caused by volumetric representation and 3D convolution: Holistic 2D prediction and Local 3D prediction. In the first stage, Planimetric Network (P-Net) estimates per-pixel likelihood for each body joint in the holistic 2D space. In the second stage, Volumetric Network (V-Net) estimates the per-voxel likelihood of each body joints in the local 3D space around the 2D estimations of the first stage, effectively reducing the computational cost. Our model outperforms existing methods by a large margin in publicly available datasets.'
author:
- |
    Gyeongsik Moon\
    ASRI, Seoul National University\
    [mks0601@snu.ac.kr]{}
- |
    Ju Yong Chang\
    Kwangwoon University\
    [juyong.chang@gmail.com]{}
- |
    Yumin Suh\
    ASRI, Seoul National University\
    [n12345@snu.ac.kr]{}
- |
    Kyoung Mu Lee\
    ASRI, Seoul National University\
    [kyoungmu@snu.ac.kr]{}
title: Holistic Planimetric prediction to Local Volumetric prediction for 3D Human Pose Estimation
---

[10]{}=-1pt

M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In [*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*]{}, pages 3686–3693, 2014.

L. Bourdev and J. Malik. Poselets: Body part detectors trained using 3d human pose annotations. In [*Computer Vision, 2009 IEEE 12th International Conference on*]{}, pages 1365–1372. IEEE, 2009.

A. Bulat and G. Tzimiropoulos. Human pose estimation via convolutional part heatmap regression. In [*European Conference on Computer Vision*]{}, pages 717–732. Springer, 2016.

J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik. Human pose estimation with iterative error feedback. In [*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*]{}, pages 4733–4742, 2016.

R. Collobert, K. Kavukcuoglu, and C. Farabet. Torch7: A matlab-like environment for machine learning. In [*BigLearn, NIPS Workshop*]{}, number EPFL-CONF-192376, 2011.

P. Felzenszwalb, D. McAllester, and D. Ramanan. A discriminatively trained, multiscale, deformable part model. In [*Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on*]{}, pages 1–8. IEEE, 2008.

V. Ganapathi, C. Plagemann, D. Koller, and S. Thrun. Real-time human pose tracking from range data. In [*European conference on computer vision*]{}, pages 738–751. Springer, 2012.

R. Girshick, J. Shotton, P. Kohli, A. Criminisi, and A. Fitzgibbon. Efficient regression of general-activity human poses from depth images. In [*2011 International Conference on Computer Vision*]{}, pages 415–422. IEEE, 2011.

D. Grest, J. Woetzel, and R. Koch. Nonlinear body pose estimation from depth images. In [*Joint Pattern Recognition Symposium*]{}, pages 285–292. Springer, 2005.

A. Haque, B. Peng, Z. Luo, A. Alahi, S. Yeung, and L. Fei-Fei. Towards viewpoint invariant 3d human pose estimation. In [*European Conference on Computer Vision*]{}, pages 160–177. Springer, 2016.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In [*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*]{}, pages 770–778, 2016.

T. Helten, A. Baak, G. Bharaj, M. M[ü]{}ller, H.-P. Seidel, and C. Theobalt. Personalization and evaluation of a real-time depth-based full body tracker. In [*3DTV-Conference, 2013 International Conference on*]{}, pages 279–286. IEEE, 2013.

H. Y. Jung, Y. Suh, G. Moon, and K. M. Lee. A sequential approach to 3d human pose estimation: Separation of localization and identification of body joints. In [*European Conference on Computer Vision*]{}, pages 747–761. Springer, 2016.

S. Knoop, S. Vacek, and R. Dillmann. Sensor fusion for 3d human body tracking with an articulated 3d body model. In [*Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on*]{}, pages 1686–1691. IEEE, 2006.

W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox detector. In [*European Conference on Computer Vision*]{}, pages 21–37. Springer, 2016.

J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In [*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*]{}, pages 3431–3440, 2015.

D. Maturana and S. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In [*Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on*]{}, pages 922–928. IEEE, 2015.

H. Moravec and A. Elfes. High resolution maps from wide angle sonar. In [*Robotics and Automation. Proceedings. 1985 IEEE International Conference on*]{}, volume 2, pages 116–121. IEEE, 1985.

A. Newell, K. Yang, and J. Deng. Stacked hourglass networks for human pose estimation. In [*European Conference on Computer Vision*]{}, pages 483–499. Springer, 2016.

G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis. Coarse-to-fine volumetric prediction for single-image 3d human pose. , 2016.

S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In [*Advances in neural information processing systems*]{}, pages 91–99, 2015.

J. Romero, H. Kjellstr[ö]{}m, and D. Kragic. Monocular real-time 3d articulated hand pose estimation. In [*2009 9th IEEE-RAS International Conference on Humanoid Robots*]{}, pages 87–92. IEEE, 2009.

J. Shotton, T. Sharp, A. Kipman, A. Fitzgibbon, M. Finocchio, A. Blake, M. Cook, and R. Moore. Real-time human pose recognition in parts from single depth images. , 56(1):116–124, 2013.

S. Song and J. Xiao. Deep sliding shapes for amodal 3d object detection in rgb-d images. In [*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*]{}, pages 808–816, 2016.

S. Thrun. Learning occupancy grid maps with forward sensor models. , 15(2):111–127, 2003.

J. J. Tompson, A. Jain, Y. LeCun, and C. Bregler. Joint training of a convolutional network and a graphical model for human pose estimation. In [*Advances in neural information processing systems*]{}, pages 1799–1807, 2014.

A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural networks. In [*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*]{}, pages 1653–1660, 2014.

S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh. Convolutional pose machines. In [*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*]{}, pages 4724–4732, 2016.

Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3d shapenets: A deep representation for volumetric shapes. In [*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*]{}, pages 1912–1920, 2015.

Y. Yang and D. Ramanan. Articulated human detection with flexible mixtures of parts. , 35(12):2878–2890, 2013.

M. Ye and R. Yang. Real-time simultaneous pose and shape estimation for articulated objects using a single depth camera. In [*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*]{}, pages 2345–2352, 2014.

H. Yub Jung, S. Lee, Y. Seok Heo, and I. Dong Yun. Random tree walk toward instantaneous 3d human pose estimation. In [*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*]{}, pages 2467–2474, 2015.
