  Symbols representing abstract states such as "dish in dishwasher" or "cup on
table" allow robots to reason over long horizons by hiding details unnecessary
for high-level planning. Current methods for learning to identify symbolic
states in visual data require large amounts of labeled training data, but
manually annotating such datasets is prohibitively expensive due to the
combinatorial number of predicates in images. We propose a novel method for
automatically labeling symbolic states in large-scale video activity datasets
by exploiting known pre- and post-conditions of actions. This automatic
labeling scheme only requires weak supervision in the form of an action label
that describes which action is demonstrated in each video. We use our framework
to train predicate classifiers to identify symbolic relationships between
objects when prompted with object bounding boxes, and demonstrate that such
predicate classifiers can match the performance of those trained with full
supervision at a fraction of the labeling cost. We also apply our framework to
an existing large-scale human activity dataset, and demonstrate the ability of
these predicate classifiers trained on human data to enable closed-loop task
planning in the real world.
