  We study roles of electron correlations on topological insulators on the
honeycomb lattice with the spin-orbit interaction. Accurate variational Monte
Carlo calculations show that the increasing on-site Coulomb interactions cause
a strong suppression of the charge Drude weight in the helical-edge metallic
states leading to a presumable Mott transition from a conventional topological
insulator to an edge Mott insulator before a transition to a bulk
antiferromagnetic insulator. The intermediate bulk-topological and
edge-Mott-insulator phase has a helical spin-liquid character with the
protected time-reversal symmetry.
