  We propose LASER, a neuro-symbolic approach to learn semantic video
representations that capture rich spatial and temporal properties in video data
by leveraging high-level logic specifications. In particular, we formulate the
problem in terms of alignment between raw videos and spatio-temporal logic
specifications. The alignment algorithm leverages a differentiable symbolic
reasoner and a combination of contrastive, temporal, and semantics losses. It
effectively and efficiently trains low-level perception models to extract a
fine-grained video representation in the form of a spatio-temporal scene graph
that conforms to the desired high-level specification. To practically reduce
the manual effort of obtaining ground truth labels, we derive logic
specifications from captions by employing a large language model with a generic
prompting template. In doing so, we explore a novel methodology that weakly
supervises the learning of spatio-temporal scene graphs with widely accessible
video-caption data. We evaluate our method on three datasets with rich spatial
and temporal specifications: 20BN-Something-Something, MUGEN, and OpenPVSG. We
demonstrate that our method learns better fine-grained video semantics than
existing baselines.
