  We show that by gauging the Schr\"odinger algebra with critical exponent $z$
and imposing suitable curvature constraints, that make diffeomorphisms
equivalent to time and space translations, one obtains a geometric structure
known as (twistless) torsional Newton-Cartan geometry (TTNC). This is a version
of torsional Newton-Cartan geometry (TNC) in which the timelike vielbein
$\tau_\mu$ must be hypersurface orthogonal. For $z=2$ this version of TTNC
geometry is very closely related to the one appearing in holographic duals of
$z=2$ Lifshitz space-times based on Einstein gravity coupled to massive vector
fields in the bulk. For $z\neq 2$ there is however an extra degree of freedom
$b_0$ that does not appear in the holographic setup. We show that the result of
the gauging procedure can be extended to include a St\"uckelberg scalar $\chi$
that shifts under the particle number generator of the Schr\"odinger algebra,
as well as an extra special conformal symmetry that allows one to gauge away
$b_0$. The resulting version of TTNC geometry is the one that appears in the
holographic setup. This shows that Schr\"odinger symmetries play a crucial role
in holography for Lifshitz space-times and that in fact the entire boundary
geometry is dictated by local Schr\"odinger invariance. Finally we show how to
extend the formalism to generic torsional Newton-Cartan geometries by relaxing
the hypersurface orthogonality condition for the timelike vielbein $\tau_\mu$.
