  We present Ego-Only, the first approach that enables state-of-the-art action
detection on egocentric (first-person) videos without any form of exocentric
(third-person) transferring. Despite the content and appearance gap separating
the two domains, large-scale exocentric transferring has been the default
choice for egocentric action detection. This is because prior works found that
egocentric models are difficult to train from scratch and that transferring
from exocentric representations leads to improved accuracy. However, in this
paper, we revisit this common belief. Motivated by the large gap separating the
two domains, we propose a strategy that enables effective training of
egocentric models without exocentric transferring. Our Ego-Only approach is
simple. It trains the video representation with a masked autoencoder finetuned
for temporal segmentation. The learned features are then fed to an
off-the-shelf temporal action localization method to detect actions. We find
that this renders exocentric transferring unnecessary by showing remarkably
strong results achieved by this simple Ego-Only approach on three established
egocentric video datasets: Ego4D, EPIC-Kitchens-100, and Charades-Ego. On both
action detection and action recognition, Ego-Only outperforms previous best
exocentric transferring methods that use orders of magnitude more labels.
Ego-Only sets new state-of-the-art results on these datasets and benchmarks
without exocentric data.
