  The electronic structure at the interface between a topological band
insulator and a Mott insulator is studied within layer dynamical mean field
theory. To represent the bulk phases of these systems, we use the generalized
Bernevig-Hughes-Zhang model that incorporates the Hubbard-like onsite Coulomb
energy U in addition to the spin-orbit coupling term that causes band
inversion. The topological and Mott insulating phases are realized by
appropriately choosing smaller and larger values of U, respectively. As
expected, the interface is found to be metallic because of the localized edge
state. When the Coulomb energy in the Mott insulator is close to the critical
value, however, this edge state exhibits its largest amplitude deep within the
Mott insulator rather than at the interface. This finding corresponds to a new
type of proximity effect induced by the neighboring topological band insulator
and demonstrates that, as a result of spin-orbit coupling within the Mott
insulator, several layers near the interface convert from the Mott insulating
phase to a topological insulating phase. Moreover, we argue that the ordinary
proximity effect, whereby a Kondo peak is induced in a Mott insulator by
neighboring metallic states, is accompanied by an additional reverse proximity
effect, by which the Kondo peak gives rise to an enhancement of the density of
states in the neighboring metallic layer.
