  Hyperbolic space can naturally embed hierarchies, unlike Euclidean space.
Hyperbolic Neural Networks (HNNs) exploit such representational power by
lifting Euclidean features into hyperbolic space for classification,
outperforming Euclidean neural networks (ENNs) on datasets with known semantic
hierarchies. However, HNNs underperform ENNs on standard benchmarks without
clear hierarchies, greatly restricting HNNs' applicability in practice.
  Our key insight is that HNNs' poorer general classification performance
results from vanishing gradients during backpropagation, caused by their hybrid
architecture connecting Euclidean features to a hyperbolic classifier. We
propose an effective solution by simply clipping the Euclidean feature
magnitude while training HNNs.
  Our experiments demonstrate that clipped HNNs become super-hyperbolic
classifiers: They are not only consistently better than HNNs which already
outperform ENNs on hierarchical data, but also on-par with ENNs on MNIST,
CIFAR10, CIFAR100 and ImageNet benchmarks, with better adversarial robustness
and out-of-distribution detection.
