  Recognizing when people have false beliefs is crucial for understanding their
actions. We introduce the novel problem of identifying when people in abstract
scenes have incorrect beliefs. We present a dataset of scenes, each visually
depicting an 8-frame story in which a character has a mistaken belief. We then
create a representation of characters' beliefs for two tasks in human action
understanding: predicting who is mistaken, and when they are mistaken.
Experiments suggest that our method for identifying mistaken characters
performs better on these tasks than simple baselines. Diagnostics on our model
suggest it learns important cues for recognizing mistaken beliefs, such as
gaze. We believe models of people's beliefs will have many applications in
action understanding, robotics, and healthcare.
