  Humor is an integral part of human lives. Despite being tremendously
impactful, it is perhaps surprising that we do not have a detailed
understanding of humor yet. As interactions between humans and AI systems
increase, it is imperative that these systems are taught to understand
subtleties of human expressions such as humor. In this work, we are interested
in the question - what content in a scene causes it to be funny? As a first
step towards understanding visual humor, we analyze the humor manifested in
abstract scenes and design computational models for them. We collect two
datasets of abstract scenes that facilitate the study of humor at both the
scene-level and the object-level. We analyze the funny scenes and explore the
different types of humor depicted in them via human studies. We model two tasks
that we believe demonstrate an understanding of some aspects of visual humor.
The tasks involve predicting the funniness of a scene and altering the
funniness of a scene. We show that our models perform well quantitatively, and
qualitatively through human studies. Our datasets are publicly available.
