  Various heuristic objectives for modeling hand-object interaction have been
proposed in past work. However, due to the lack of a cohesive framework, these
objectives often possess a narrow scope of applicability and are limited by
their efficiency or accuracy. In this paper, we propose HandyPriors, a unified
and general pipeline for pose estimation in human-object interaction scenes by
leveraging recent advances in differentiable physics and rendering. Our
approach employs rendering priors to align with input images and segmentation
masks along with physics priors to mitigate penetration and relative-sliding
across frames. Furthermore, we present two alternatives for hand and object
pose estimation. The optimization-based pose estimation achieves higher
accuracy, while the filtering-based tracking, which utilizes the differentiable
priors as dynamics and observation models, executes faster. We demonstrate that
HandyPriors attains comparable or superior results in the pose estimation task,
and that the differentiable physics module can predict contact information for
pose refinement. We also show that our approach generalizes to perception
tasks, including robotic hand manipulation and human-object pose estimation in
the wild.
