  This paper reviews concepts, modeling approaches, and recent findings along a
spectrum of different levels of abstraction of neural network models including
generalization across (1) Samples, (2) Distributions, (3) Domains, (4) Tasks,
(5) Modalities, and (6) Scopes. Strategies for (1) sample generalization from
training to test data are discussed, with suggestive evidence presented that,
at least for the ImageNet dataset, popular classification models show
substantial overfitting. An empirical example and perspectives from statistics
highlight how models' (2) distribution generalization can benefit from
consideration of causal relationships and counterfactual scenarios. Transfer
learning approaches and results for (3) domain generalization are summarized,
as is the wealth of domain generalization benchmark datasets available. Recent
breakthroughs surveyed in (4) task generalization include few-shot
meta-learning approaches and the emergence of transformer-based foundation
models such as those used for language processing. Studies performing (5)
modality generalization are reviewed, including those that integrate image and
text data and that apply a biologically-inspired network across olfactory,
visual, and auditory modalities. Higher-level (6) scope generalization results
are surveyed, including graph-based approaches to represent symbolic knowledge
in networks and attribution strategies for improving networks' explainability.
Additionally, concepts from neuroscience are discussed on the modular
architecture of brains and the steps by which dopamine-driven conditioning
leads to abstract thinking.
