  Representing graphs as sets of node embeddings in certain curved Riemannian
manifolds has recently gained momentum in machine learning due to their
desirable geometric inductive biases, e.g., hierarchical structures benefit
from hyperbolic geometry. However, going beyond embedding spaces of constant
sectional curvature, while potentially more representationally powerful, proves
to be challenging as one can easily lose the appeal of computationally
tractable tools such as geodesic distances or Riemannian gradients. Here, we
explore computationally efficient matrix manifolds, showcasing how to learn and
optimize graph embeddings in these Riemannian spaces. Empirically, we
demonstrate consistent improvements over Euclidean geometry while often
outperforming hyperbolic and elliptical embeddings based on various metrics
that capture different graph properties. Our results serve as new evidence for
the benefits of non-Euclidean embeddings in machine learning pipelines.
