  The joint-embedding predictive architecture (JEPA) recently has shown
impressive results in extracting visual representations from unlabeled imagery
under a masking strategy. However, we reveal its disadvantages, notably its
insufficient understanding of local semantics. This deficiency originates from
masked modeling in the embedding space, resulting in a reduction of
discriminative power and can even lead to the neglect of critical local
semantics. To bridge this gap, we introduce DMT-JEPA, a novel masked modeling
objective rooted in JEPA, specifically designed to generate discriminative
latent targets from neighboring information. Our key idea is simple: we
consider a set of semantically similar neighboring patches as a target of a
masked patch. To be specific, the proposed DMT-JEPA (a) computes feature
similarities between each masked patch and its corresponding neighboring
patches to select patches having semantically meaningful relations, and (b)
employs lightweight cross-attention heads to aggregate features of neighboring
patches as the masked targets. Consequently, DMT-JEPA demonstrates strong
discriminative power, offering benefits across a diverse spectrum of downstream
tasks. Through extensive experiments, we demonstrate our effectiveness across
various visual benchmarks, including ImageNet-1K image classification, ADE20K
semantic segmentation, and COCO object detection tasks. Code is available at:
\url{https://github.com/DMTJEPA/DMTJEPA}.
