  The study of the (logarithm of the) {\em fidelity} i.e., of the overlap
amplitude, between ground states of Hamiltonians corresponding to different
coupling constants, provides a valuable insight on critical phenomena. When the
parameters are infinitesimally close, it is known that the leading term behaves
as $O(L^\alpha)$ ($L$ system size) where $\alpha$ is equal to the spatial
dimension $d$ for gapped systems, and otherwise depends on the critical
exponents. Here we show that when parameters are changed along a critical
manifold, a sub-leading O(1) term can appear. This term, somewhat similar to
the topological entanglement entropy, depends only on the system's universality
class and encodes non-trivial information about the topology of the system. We
relate it to universal $g$ factors and partition functions of (boundary)
conformal field theory in $d=1$ and $d=2$ dimensions. Numerical checks are
presented on the simple example of the XXZ chain.
