  Three-dimensional gapless semiconductors with quadratic band touching, such
as HgTe, $\alpha$-Sn, or Pr$_2$Ir$_2$O$_7$ are believed to display a
non-Fermi-liquid ground state due to long-range electron-electron interaction.
We argue that this state is inherently unstable towards spontaneous formation
of a (topological) excitonic insulator. The instability can be parameterized by
a critical fermion number $N_c$. For $N < N_c$ the rotational symmetry is
spontaneously broken, the system develops a gap in the spectrum, and features a
finite nematic order parameter. To leading order in the 1/N expansion and in
the static approximation, the analogy with the problem of dynamical mass
generation in (2+1)-dimensional quantum electrodynamics yields $N_c =
16/[3\pi(\pi-2)]$. Taking the important dynamical screening effects into
account, we find that $N_c \geq 2.6(2)$ and therefore safely above the physical
value of $N = 1$. Some experimental consequences of the nematic ground state
are discussed.
