  Embodied AI models often employ off the shelf vision backbones like CLIP to
encode their visual observations. Although such general purpose representations
encode rich syntactic and semantic information about the scene, much of this
information is often irrelevant to the specific task at hand. This introduces
noise within the learning process and distracts the agent's focus from
task-relevant visual cues. Inspired by selective attention in humans-the
process through which people filter their perception based on their
experiences, knowledge, and the task at hand-we introduce a parameter-efficient
approach to filter visual stimuli for embodied AI. Our approach induces a
task-conditioned bottleneck using a small learnable codebook module. This
codebook is trained jointly to optimize task reward and acts as a
task-conditioned selective filter over the visual observation. Our experiments
showcase state-of-the-art performance for object goal navigation and object
displacement across 5 benchmarks, ProcTHOR, ArchitecTHOR, RoboTHOR, AI2-iTHOR,
and ManipulaTHOR. The filtered representations produced by the codebook are
also able generalize better and converge faster when adapted to other
simulation environments such as Habitat. Our qualitative analyses show that
agents explore their environments more effectively and their representations
retain task-relevant information like target object recognition while ignoring
superfluous information about other objects. Code and pretrained models are
available at our project website: https://embodied-codebook.github.io.
