  The detection of the (semi)metal-insulator phase transition can be extremely
difficult if the local order parameter which characterizes the ordered phase is
unknown.In some cases, it is even impossible to define a local order parameter:
the most prominent example of such system is the spin liquid state. This state
was proposed to exist in theHubbard model on the hexagonal lattice in a region
between the semimetal phase and the antiferromagnetic insulator phase. The
existence of this phase has been the subject of a long debate. In order to
detect these exotic phases we must use alternative methods to those used for
more familiar examples of spontaneous symmetry breaking. We have modified the
Backus-Gilbert method of analytic continuation which was previously used in the
calculation of the pion quasiparticle mass in lattice QCD. The modification of
the method consists of the introduction of the Tikhonov regularization scheme
which was used to treat the ill-conditioned kernel. This modified
Backus-Gilbert method is applied to the Euclidean propagators in momentum space
calculated using the hybridMonte Carlo algorithm. In this way, it is possible
to reconstruct the full dispersion relation and to estimate the mass gap, which
is a direct signal of the transition to the insulating state. We demonstrate
the utility of this method in our calculations for the Hubbard model on the
hexagonal lattice. We also apply the method to the metal-insulator phase
transition in the Hubbard-Coulomb model on the square lattice.
