  In many neural networks, different values of the parameters may result in the
same loss value. Parameter space symmetries are loss-invariant transformations
that change the model parameters. Teleportation applies such transformations to
accelerate optimization. However, the exact mechanism behind this algorithm's
success is not well understood. In this paper, we show that teleportation not
only speeds up optimization in the short-term, but gives overall faster time to
convergence. Additionally, teleporting to minima with different curvatures
improves generalization, which suggests a connection between the curvature of
the minimum and generalization ability. Finally, we show that integrating
teleportation into a wide range of optimization algorithms and
optimization-based meta-learning improves convergence. Our results showcase the
versatility of teleportation and demonstrate the potential of incorporating
symmetry in optimization.
