  Identifying specific anatomical structures (\textit{e.g.}, lesions or
landmarks) in medical images plays a fundamental role in medical image
analysis. Exemplar-based landmark detection methods are receiving increasing
attention since they can detect arbitrary anatomical points in inference while
do not need landmark annotations in training. They use self-supervised learning
to acquire a discriminative embedding for each voxel within the image. These
approaches can identify corresponding landmarks through nearest neighbor
matching and has demonstrated promising results across various tasks. However,
current methods still face challenges in: (1) differentiating voxels with
similar appearance but different semantic meanings (\textit{e.g.}, two adjacent
structures without clear borders); (2) matching voxels with similar semantics
but markedly different appearance (\textit{e.g.}, the same vessel before and
after contrast injection); and (3) cross-modality matching (\textit{e.g.},
CT-MRI landmark-based registration). To overcome these challenges, we propose
universal anatomical embedding (UAE), which is a unified framework designed to
learn appearance, semantic, and cross-modality anatomical embeddings.
Specifically, UAE incorporates three key innovations: (1) semantic embedding
learning with prototypical contrastive loss; (2) a fixed-point-based matching
strategy; and (3) an iterative approach for cross-modality embedding learning.
We thoroughly evaluated UAE across intra- and inter-modality tasks, including
one-shot landmark detection, lesion tracking on longitudinal CT scans, and
CT-MRI affine/rigid registration with varying field of view. Our results
suggest that UAE outperforms state-of-the-art methods, offering a robust and
versatile approach for landmark based medical image analysis tasks. Code and
trained models are available at: \href{https://shorturl.at/bgsB3}
