  Most prior art in visual understanding relies solely on analyzing the "what"
(e.g., event recognition) and "where" (e.g., event localization), which in some
cases, fails to describe correct contextual relationships between events or
leads to incorrect underlying visual attention. Part of what defines us as
human and fundamentally different from machines is our instinct to seek
causality behind any association, say an event Y that happened as a direct
result of event X. To this end, we propose iPerceive, a framework capable of
understanding the "why" between events in a video by building a common-sense
knowledge base using contextual cues to infer causal relationships between
objects in the video. We demonstrate the effectiveness of our technique using
the dense video captioning (DVC) and video question answering (VideoQA) tasks.
Furthermore, while most prior work in DVC and VideoQA relies solely on visual
information, other modalities such as audio and speech are vital for a human
observer's perception of an environment. We formulate DVC and VideoQA tasks as
machine translation problems that utilize multiple modalities. By evaluating
the performance of iPerceive DVC and iPerceive VideoQA on the ActivityNet
Captions and TVQA datasets respectively, we show that our approach furthers the
state-of-the-art. Code and samples are available at: iperceive.amanchadha.com.
