  While Rotary Position Embeddings (RoPE) for natural language performs well
and has become widely adopted, its adoption for other modalities has been
slower. Here, we introduce Lie group Relative position Encodings (LieRE) that
goes beyond RoPE in supporting higher dimensional inputs. We evaluate the
performance of LieRE on 2D and 3D image classification tasks and observe that
LieRE leads to marked improvements in performance (up to 6%), training
efficiency (3.5x reduction), data efficiency (30%) compared to the baselines of
RoFormer, DeiT III, RoPE-Mixed and Vision-Llama
