  While pre-trained large-scale vision models have shown significant promise
for semantic correspondence, their features often struggle to grasp the
geometry and orientation of instances. This paper identifies the importance of
being geometry-aware for semantic correspondence and reveals a limitation of
the features of current foundation models under simple post-processing. We show
that incorporating this information can markedly enhance semantic
correspondence performance with simple but effective solutions in both
zero-shot and supervised settings. We also construct a new challenging
benchmark for semantic correspondence built from an existing animal pose
estimation dataset, for both pre-training validating models. Our method
achieves a<EMAIL_ADDRESS>score of 65.4 (zero-shot) and 85.6 (supervised) on the
challenging SPair-71k dataset, outperforming the state of the art by 5.5p and
11.0p absolute gains, respectively. Our code and datasets are publicly
available at: https://telling-left-from-right.github.io/.
