  We introduce hyperbolic attention networks to endow neural networks with
enough capacity to match the complexity of data with hierarchical and power-law
structure. A few recent approaches have successfully demonstrated the benefits
of imposing hyperbolic geometry on the parameters of shallow networks. We
extend this line of work by imposing hyperbolic geometry on the activations of
neural networks. This allows us to exploit hyperbolic geometry to reason about
embeddings produced by deep networks. We achieve this by re-expressing the
ubiquitous mechanism of soft attention in terms of operations defined for
hyperboloid and Klein models. Our method shows improvements in terms of
generalization on neural machine translation, learning on graphs and visual
question answering tasks while keeping the neural representations compact.
