  Observations in quantum weak measurements are determined by complex numbers
called weak values. We present a geometrical interpretation of the argument of
weak values of general Hermitian observables in $N$-dimensional quantum systems
in terms of geometric phases. We formulate an arbitrary weak value in function
of three real vectors on the unit sphere in $N^2-1$ dimensions, $S^{N^2-2}$.
These vectors are linked to the initial and final states, and to the weakly
measured observable, respectively. We express pure states in the complex
projective space of $N-1$ dimensions, $\mathbb{C}\textrm{P}^{N-1}$, which has a
non-trivial representation as a $2N-2$ dimensional submanifold of $S^{N^2-2}$
(a generalization of the Bloch sphere for qudits). The argument of the weak
value of a projector on a pure state of an $N$-level quantum system describes a
geometric phase associated to the symplectic area of the geodesic triangle
spanned by the vectors representing the pre-selected state, the projector and
the post-selected state in $\mathbb{C}\textrm{P}^{N-1}$. We then proceed to
show that the argument of the weak value of a general observable is equivalent
to the argument of an effective Bargmann invariant. Hence, we extend the
geometrical interpretation of projector weak values to weak values of general
observables. In particular, we consider the generators of SU($N$) given by the
generalized Gell-Mann matrices. Finally, we study in detail the case of the
argument of weak values of general observables in two-level systems and we
illustrate weak measurements in larger dimensional systems by considering
projectors on degenerate subspaces, as well as Hermitian quantum gates.
