  Recent discoveries have spurred the theoretical prediction and experimental
realization of novel materials that have topological properties arising from
band inversion. Such topological insulators are insulating in the bulk but have
conductive surface or edge states. Topological materials show various unusual
physical properties and are surmised to enable the creation of exotic
Majorana-fermion quasiparticles. How the signatures of topological behavior
evolve when the system size is reduced is interesting from both a fundamental
and an application-oriented point of view, as such understanding may form the
basis for tailoring systems to be in specific topological phases. This work
considers the specific case of quantum-well confinement defining
two-dimensional layers. Based on the effective-Hamiltonian description of bulk
topological insulators, and using a harmonic-oscillator potential as an example
for a softer-than-hard-wall confinement, we have studied the interplay of band
inversion and size quantization. Our model system provides a useful platform
for systematic study of the transition between the normal and topological
phases, including the development of band inversion and the formation of
massless-Dirac-fermion surface states. The effects of bare size quantization,
two-dimensional-subband mixing, and electron-hole asymmetry are disentangled
and their respective physical consequences elucidated.
