  One-dimensional (1D) edge states formed at the boundaries of 2D normal and
topological insulators have shown intriguing quantum phases such as charge
density wave and quantum spin Hall effect. Based on first-principles
density-functional theory calculations including spin-orbit coupling (SOC), we
show that the edge states of zigzag Sb(111) and Bi(111) nanoribbons drastically
change the stability of their edge structures. For zigzag Sb(111) nanoribbon,
the Peierls-distorted or reconstructed edge structure is stabilized by a
band-gap opening. However, for zigzag Bi(111) nanoribbon, such two insulating
structures are destabilized due to the presence of topologically protected
gapless edge states, resulting in the stabilization of a metallic,
shear-distorted edge structure. We also show that the edge states of the
Bi(111) nanoribbon exhibit a larger Rashba-type spin splitting at the boundary
of Brillouin zone, compared to those of the Sb(111) nanoribbon. Interestingly,
the spin textures of edge states in the Peierls-distorted Sb edge structure and
the shear-distorted Bi edge structure have all three spin components
perpendicular and parallel to the edges, due to their broken mirror-plane
symmetry. The present findings demonstrate that the topologically trivial and
nontrivial edge states play crucial roles in determining the edge structures of
normal and topological insulators.
