  The emotional theory of mind problem requires facial expressions, body pose,
contextual information and implicit commonsense knowledge to reason about the
person's emotion and its causes, making it currently one of the most difficult
problems in affective computing. In this work, we propose multiple methods to
incorporate the emotional reasoning capabilities by constructing "narrative
captions" relevant to emotion perception, that includes contextual and physical
signal descriptors that focuses on "Who", "What", "Where" and "How" questions
related to the image and emotions of the individual. We propose two distinct
ways to construct these captions using zero-shot classifiers (CLIP) and
fine-tuning visual-language models (LLaVA) over human generated descriptors. We
further utilize these captions to guide the reasoning of language (GPT-4) and
vision-language models (LLaVa, GPT-Vision). We evaluate the use of the
resulting models in an image-to-language-to-emotion task. Our experiments
showed that combining the "Fast" narrative descriptors and "Slow" reasoning of
language models is a promising way to achieve emotional theory of mind.
