  Learning commonsense reasoning from visual contexts and scenes in real-world
is a crucial step toward advanced artificial intelligence. However, existing
video reasoning benchmarks are still inadequate since they were mainly designed
for factual or situated reasoning and rarely involve broader knowledge in the
real world. Our work aims to delve deeper into reasoning evaluations,
specifically within dynamic, open-world, and structured context knowledge. We
propose a new benchmark (SOK-Bench), consisting of 44K questions and 10K
situations with instance-level annotations depicted in the videos. The
reasoning process is required to understand and apply situated knowledge and
general knowledge for problem-solving. To create such a dataset, we propose an
automatic and scalable generation method to generate question-answer pairs,
knowledge graphs, and rationales by instructing the combinations of LLMs and
MLLMs. Concretely, we first extract observable situated entities, relations,
and processes from videos for situated knowledge and then extend to open-world
knowledge beyond the visible content. The task generation is facilitated
through multiple dialogues as iterations and subsequently corrected and refined
by our designed self-promptings and demonstrations. With a corpus of both
explicit situated facts and implicit commonsense, we generate associated
question-answer pairs and reasoning processes, finally followed by manual
reviews for quality assurance. We evaluated recent mainstream large
vision-language models on the benchmark and found several insightful
conclusions. For more information, please refer to our benchmark at
www.bobbywu.com/SOKBench.
