  We investigate correlation effects in two dimensional topological insulators
(TI). In the first part, we discuss finite size effects for interacting systems
of different sizes in a ribbon geometry. For large systems, there are two pairs
of well separated massless modes on both edges. For these systems, we analyze
the finite size effects using a standard bosonization approach. For small
systems, where the edge states are massive Dirac fermions, we use the
inhomogeneous dynamical mean field theory (DMFT) combined with iterative
perturbation theory as an impurity solver to study interaction effects. We show
that the finite size gap in the edge states is renormalized for weak
interactions, which is consistent with a Fermi-liquid picture for small size
TIs. In the second part, we investigate phase transitions in finite size TIs at
zero temperature focusing on the effects of possible inter-edge Umklapp
scattering for the edge states within the inhomogeneous DMFT using the
numerical renormalization group. We show that correlation effects are
effectively stronger near the edge sites because the coordination number is
smaller than in the bulk. Therefore, the localization of the edge states around
the edge sites, which is a fundamental property in TIs, is weakened for strong
coupling strengths. However, we find no signs for "edge Mott insulating states"
and the system stays in the topological insulating state, which is
adiabatically connected to the non-interacting state, for all interaction
strengths smaller than the critical value. Increasing the interaction further,
a nearly homogeneous Mott insulating state is stabilized.
