  We study a crucial yet often overlooked issue inherent to Vision Transformers
(ViTs): feature maps of these models exhibit grid-like artifacts, which hurt
the performance of ViTs in downstream dense prediction tasks such as semantic
segmentation, depth prediction, and object discovery. We trace this issue down
to the positional embeddings at the input stage. To mitigate this, we propose a
two-stage denoising approach, termed Denoising Vision Transformers (DVT). In
the first stage, we separate the clean features from those contaminated by
positional artifacts by enforcing cross-view feature consistency with neural
fields on a per-image basis. This per-image optimization process extracts
artifact-free features from raw ViT outputs, providing clean feature estimates
for offline applications. In the second stage, we train a lightweight
transformer block to predict clean features from raw ViT outputs, leveraging
the derived estimates of the clean features as supervision. Our method, DVT,
does not require re-training the existing pre-trained ViTs, and is immediately
applicable to any Vision Transformer architecture. We evaluate our method on a
variety of representative ViTs (DINO, DeiT-III, EVA02, CLIP, DINOv2,
DINOv2-reg) and demonstrate that DVT consistently improves existing
state-of-the-art general-purpose models in semantic and geometric tasks across
multiple datasets. We hope our study will encourage a re-evaluation of ViT
design, especially regarding the naive use of positional embeddings. Our code
and checkpoints are publicly available.
