  Biological vision systems make adaptive use of context to recognize objects
in new settings with novel contexts as well as occluded or blurry objects in
familiar settings. In this paper, we investigate how vision models adaptively
use context for out-of-distribution (OOD) generalization and leverage our
analysis results to improve model OOD generalization. First, we formulate two
distinct OOD settings where the contexts are either irrelevant
(Background-Invariance) or beneficial (Object-Disambiguation), reflecting the
diverse contextual challenges faced in biological vision. We then analyze model
performance in these two different OOD settings and demonstrate that models
that excel in one setting tend to struggle in the other. Notably, prior works
on learning causal features improve on one setting but hurt in the other. This
underscores the importance of generalizing across both OOD settings, as this
ability is crucial for both human cognition and robust AI systems. Next, to
better understand the model properties contributing to OOD generalization, we
use representational geometry analysis and our own probing methods to examine a
population of models, and we discover that those with more factorized
representations and appropriate feature weighting are more successful in
handling Background-Invariance and Object-Disambiguation tests. We further
validate these findings through causal intervention on representation
factorization and feature weighting to demonstrate their causal effect on
performance. Lastly, we propose new augmentation methods to enhance model
generalization. These methods outperform strong baselines, yielding
improvements in both in-distribution and OOD tests. In conclusion, to replicate
the generalization abilities of biological vision, computer vision models must
have factorized object vs. background representations and appropriately weight
both kinds of features.
