  Building perceptual systems for robotics which perform well under tight
computational budgets requires novel architectures which rethink the
traditional computer vision pipeline. Modern vision architectures require the
agent to build a summary representation of the entire scene, even if most of
the input is irrelevant to the agent's current goal. In this work, we flip this
paradigm, by introducing EarlyFusion vision models that condition on a goal to
build custom representations for downstream tasks. We show that these goal
specific representations can be learned more quickly, are substantially more
parameter efficient, and more robust than existing attention mechanisms in our
domain. We demonstrate the effectiveness of these methods on a simulated
robotic item retrieval problem that is trained in a fully end-to-end manner via
imitation learning.
