  Existing 3D human pose estimation models suffer performance drop when
applying to new scenarios with unseen poses due to their limited
generalizability. In this work, we propose a novel framework, Inference Stage
Optimization (ISO), for improving the generalizability of 3D pose models when
source and target data come from different pose distributions. Our main insight
is that the target data, even though not labeled, carry valuable priors about
their underlying distribution. To exploit such information, the proposed ISO
performs geometry-aware self-supervised learning (SSL) on each single target
instance and updates the 3D pose model before making prediction. In this way,
the model can mine distributional knowledge about the target scenario and
quickly adapt to it with enhanced generalization performance. In addition, to
handle sequential target data, we propose an online mode for implementing our
ISO framework via streaming the SSL, which substantially enhances its
effectiveness. We systematically analyze why and how our ISO framework works on
diverse benchmarks under cross-scenario setup. Remarkably, it yields new
state-of-the-art of 83.6% 3D PCK on MPI-INF-3DHP, improving upon the previous
best result by 9.7%. Code will be released.
