  Advances in deep learning have enabled the development of models that have
exhibited a remarkable tendency to recognize and even localize actions in
videos. However, they tend to experience errors when faced with scenes or
examples beyond their initial training environment. Hence, they fail to adapt
to new domains without significant retraining with large amounts of annotated
data. In this paper, we propose to overcome these limitations by moving to an
open-world setting by decoupling the ideas of recognition and reasoning.
Building upon the compositional representation offered by Grenander's Pattern
Theory formalism, we show that attention and commonsense knowledge can be used
to enable the self-supervised discovery of novel actions in egocentric videos
in an open-world setting, where data from the observed environment (the target
domain) is open i.e., the vocabulary is partially known and training examples
(both labeled and unlabeled) are not available. We show that our approach can
infer and learn novel classes for open vocabulary classification in egocentric
videos and novel object detection with zero supervision. Extensive experiments
show its competitive performance on two publicly available egocentric action
recognition datasets (GTEA Gaze and GTEA Gaze+) under open-world conditions.
