  Modality differences have led to the development of heterogeneous
architectures for vision and language models. While images typically require 2D
non-causal modeling, texts utilize 1D causal modeling. This distinction poses
significant challenges in constructing unified multi-modal models. This paper
explores the feasibility of representing images using 1D causal modeling. We
identify an "over-focus" issue in existing 1D causal vision models, where
attention overly concentrates on a small proportion of visual tokens. The issue
of "over-focus" hinders the model's ability to extract diverse visual features
and to receive effective gradients for optimization. To address this, we
propose De-focus Attention Networks, which employ learnable bandpass filters to
create varied attention patterns. During training, large and scheduled drop
path rates, and an auxiliary loss on globally pooled features for global
understanding tasks are introduced. These two strategies encourage the model to
attend to a broader range of tokens and enhance network optimization. Extensive
experiments validate the efficacy of our approach, demonstrating that 1D causal
visual representation can perform comparably to 2D non-causal representation in
tasks such as global perception, dense prediction, and multi-modal
understanding. Code is released at
https://github.com/OpenGVLab/De-focus-Attention-Networks.
