  Recently, a correspondence has been proposed between spectral theory and
topological strings on toric Calabi-Yau manifolds. In this paper we develop in
detail this correspondence for mirror curves of higher genus, which display
many new features as compared to the genus one case studied so far. Given a
curve of genus g, our quantization scheme leads to g different trace class
operators. Their spectral properties are encoded in a generalized spectral
determinant, which is an entire function on the Calabi-Yau moduli space. We
conjecture an exact expression for this spectral determinant in terms of the
standard and refined topological string amplitudes. This conjecture provides a
non-perturbative definition of the topological string on these geometries, in
which the genus expansion emerges in a suitable 't Hooft limit of the spectral
traces of the operators. In contrast to what happens in quantum integrable
systems, our quantization scheme leads to a single quantization condition,
which is elegantly encoded by the vanishing of a quantum-deformed theta
function on the mirror curve. We illustrate our general theory by analyzing in
detail the resolved C3/Z5 orbifold, which is the simplest toric Calabi-Yau
manifold with a genus two mirror curve. By applying our conjecture to this
example, we find new quantization conditions for quantum mechanical operators,
in terms of genus two theta functions, as well as new number-theoretic
properties for the periods of this Calabi-Yau.
