  In-context learning$\unicode{x2013}$the ability to configure a model's
behavior with different prompts$\unicode{x2013}$has revolutionized the field of
natural language processing, alleviating the need for task-specific models and
paving the way for generalist models capable of assisting with any query.
Computer vision, in contrast, has largely stayed in the former regime:
specialized decoders and finetuning protocols are generally required to perform
dense tasks such as semantic segmentation and depth estimation. In this work we
explore a simple mechanism for in-context learning of such scene understanding
tasks: nearest neighbor retrieval from a prompt of annotated features. We
propose a new pretraining protocol$\unicode{x2013}$leveraging attention within
and across images$\unicode{x2013}$which yields representations particularly
useful in this regime. The resulting Hummingbird model, suitably prompted,
performs various scene understanding tasks without modification while
approaching the performance of specialists that have been finetuned for each
task. Moreover, Hummingbird can be configured to perform new tasks much more
efficiently than finetuned models, raising the possibility of scene
understanding in the interactive assistant regime.
