  The [111]-oriented InAs/GaSb and GaSb/InAs core-shell nanowires have been
studied by the $8\times 8$ Luttinger-Kohn $\vec{k}\cdot\vec{p}$ Hamiltonian to
search for non-vanishing fundamental gaps between inverted electron and hole
bands. We focus on the variations of the topologically nontrivial fundamental
gap, the hybridization gap, and the effective gap with the core radius and
shell thickness of the nanowires. The evolutions of all the energy gaps with
the structural parameters are shown to be dominantly governed by quantum size
effects. With a fixed core radius, a topologically nontrivial fundamental gap
exists only at intermediate shell thicknesses. The maximum gap is $\sim 4.4$
meV for GaSb/InAs and $\sim 3.5$ meV for InAs/GaSb core-shell nanowires, and
for the GaSb/InAs core-shell nanowires the gap persists over a wider range of
geometrical parameters. The intrinsic reason for these differences between the
two types of nanowires is that in the shell the electron-like states of InAs is
more delocalized than the hole-like state of GaSb, while in the core the
hole-like state of GaSb is more delocalized than the electron-like state of
InAs, and both features favor stronger electron-hole hybridization. Since
similar features of the electron- and hole-like states have been found in
nanowires of other materials, it could serve as a common rule to put the
hole-like state in the core while the electron-like state in the shell of a
core-shell nanowire to achieve better topological properties.
