  The purpose of this benchmark is to evaluate the planning and control aspects
of robotic in-hand manipulation systems. The goal is to assess the system's
ability to change the pose of a hand-held object by either using the fingers,
environment or a combination of both. Given an object surface mesh from the YCB
data-set, we provide examples of initial and goal states (i.e.\ static object
poses and fingertip locations) for various in-hand manipulation tasks. We
further propose metrics that measure the error in reaching the goal state from
a specific initial state, which, when aggregated across all tasks, also serves
as a measure of the system's in-hand manipulation capability. We provide
supporting software, task examples, and evaluation results associated with the
benchmark. All the supporting material is available at
https://robot-learning.cs.utah.edu/project/benchmarking_in_hand_manipulation
