  A technical note aiming to offer deeper intuition for the LayerNorm function
common in deep neural networks. LayerNorm is defined relative to a
distinguished 'neural' basis, but it does more than just normalize the
corresponding vector elements. Rather, it implements a composition -- of linear
projection, nonlinear scaling, and then affine transformation -- on input
activation vectors. We develop both a new mathematical expression and geometric
intuition, to make the net effect more transparent. We emphasize that, when
LayerNorm acts on an N-dimensional vector space, all outcomes of LayerNorm lie
within the intersection of an (N-1)-dimensional hyperplane and the interior of
an N-dimensional hyperellipsoid. This intersection is the interior of an
(N-1)-dimensional hyperellipsoid, and typical inputs are mapped near its
surface. We find the direction and length of the principal axes of this
(N-1)-dimensional hyperellipsoid via the eigen-decomposition of a simply
constructed matrix.
