  Identifying common patterns among events is a key ability in human and
machine perception, as it underlies intelligent decision making. We propose an
approach for learning semantic relational set abstractions on videos, inspired
by human learning. We combine visual features with natural language supervision
to generate high-level representations of similarities across a set of videos.
This allows our model to perform cognitive tasks such as set abstraction (which
general concept is in common among a set of videos?), set completion (which new
video goes well with the set?), and odd one out detection (which video does not
belong to the set?). Experiments on two video benchmarks, Kinetics and
Multi-Moments in Time, show that robust and versatile representations emerge
when learning to recognize commonalities among sets. We compare our model to
several baseline algorithms and show that significant improvements result from
explicitly learning relational abstractions with semantic supervision.
