  Different video understanding tasks are typically treated in isolation, and
even with distinct types of curated data (e.g., classifying sports in one
dataset, tracking animals in another). However, in wearable cameras, the
immersive egocentric perspective of a person engaging with the world around
them presents an interconnected web of video understanding tasks -- hand-object
manipulations, navigation in the space, or human-human interactions -- that
unfold continuously, driven by the person's goals. We argue that this calls for
a much more unified approach. We propose EgoTask Translation (EgoT2), which
takes a collection of models optimized on separate tasks and learns to
translate their outputs for improved performance on any or all of them at once.
Unlike traditional transfer or multi-task learning, EgoT2's flipped design
entails separate task-specific backbones and a task translator shared across
all tasks, which captures synergies between even heterogeneous tasks and
mitigates task competition. Demonstrating our model on a wide array of video
tasks from Ego4D, we show its advantages over existing transfer paradigms and
achieve top-ranked results on four of the Ego4D 2022 benchmark challenges.
