  We introduce ViLPAct, a novel vision-language benchmark for human activity
planning. It is designed for a task where embodied AI agents can reason and
forecast future actions of humans based on video clips about their initial
activities and intents in text. The dataset consists of 2.9k videos from
\charades extended with intents via crowdsourcing, a multi-choice question test
set, and four strong baselines. One of the baselines implements a neurosymbolic
approach based on a multi-modal knowledge base (MKB), while the other ones are
deep generative models adapted from recent state-of-the-art (SOTA) methods.
According to our extensive experiments, the key challenges are compositional
generalization and effective use of information from both modalities.
