  In this paper we introduce the problem of Visual Semantic Role Labeling:
given an image we want to detect people doing actions and localize the objects
of interaction. Classical approaches to action recognition either study the
task of action classification at the image or video clip level or at best
produce a bounding box around the person doing the action. We believe such an
output is inadequate and a complete understanding can only come when we are
able to associate objects in the scene to the different semantic roles of the
action. To enable progress towards this goal, we annotate a dataset of 16K
people instances in 10K images with actions they are doing and associate
objects in the scene with different semantic roles for each action. Finally, we
provide a set of baseline algorithms for this task and analyze error modes
providing directions for future work.
