  There has been tremendous recent progress in realizing topological insulator
initiated by the proposal of Kane and Mele for the graphene system. They have
suggested that the odd $Z_2$ index for the graphene manifests the spin filtered
edge states for the graphene nanoribbons, which lead to the quantum spin Hall
effect(QSHE). Here we investigate the role of the spin-orbit interaction both
for the zigzag and armchair nanoribbons with special care in the edge geometry.
For the pristine zigzag nanoribbons, we have shown that one of the $\sigma$
edge bands located near E=0 lifts up the energy of the spin filtered chiral
edge states at the zone boundary by warping the $\pi$-edge bands, and hence the
QSHE does not occur. Upon increasing the carrier density above a certain
critical value, the spin filtered edge states are formed leading to the QSHE.
We suggest that the hydrogen passivation on the edge can recover the original
feature of the QSHE. For the armchair nanoribbon, the QSHE is shown to be
stable. We have also derived the real space effective hamiltonian, which
demonstrates that the on-site energy and the effective spin orbit coupling
strength are strongly enhanced near the ribbon edges. We have shown that the
steep rise of the confinement potential thus obtained is responsible for the
warping of the $\pi$-edge bands.
