  Deep learning-based 3D human pose estimation performs best when trained on
large amounts of labeled data, making combined learning from many datasets an
important research direction. One obstacle to this endeavor are the different
skeleton formats provided by different datasets, i.e., they do not label the
same set of anatomical landmarks. There is little prior research on how to best
supervise one model with such discrepant labels. We show that simply using
separate output heads for different skeletons results in inconsistent depth
estimates and insufficient information sharing across skeletons. As a remedy,
we propose a novel affine-combining autoencoder (ACAE) method to perform
dimensionality reduction on the number of landmarks. The discovered latent 3D
points capture the redundancy among skeletons, enabling enhanced information
sharing when used for consistency regularization. Our approach scales to an
extreme multi-dataset regime, where we use 28 3D human pose datasets to
supervise one model, which outperforms prior work on a range of benchmarks,
including the challenging 3D Poses in the Wild (3DPW) dataset. Our code and
models are available for research purposes.
