  We consider the problem of understanding real world tasks depicted in visual
images. While most existing image captioning methods excel in producing natural
language descriptions of visual scenes involving human tasks, there is often
the need for an understanding of the exact task being undertaken rather than a
literal description of the scene. We leverage insights from real world task
understanding systems, and propose a framework composed of convolutional neural
networks, and an external hierarchical task ontology to produce task
descriptions from input images. Detailed experiments highlight the efficacy of
the extracted descriptions, which could potentially find their way in many
applications, including image alt text generation.
