  The structural phase transition in Ta$_2$NiSe$_5$ has been envisioned as
driven by the formation of an excitonic insulating phase. However, the role of
structural and electronic instabilities on crystal symmetry breaking has yet to
be disentangled. Meanwhile, the phase transition in its complementary material
Ta$_2$NiS$_5$ does not show any experimental hints of an excitonic insulating
phase. We present a microscopic investigation of the electronic and phononic
effects involved in the structural phase transition in Ta$_2$NiSe$_5$ and
Ta$_2$NiS$_5$ using extensive first-principles calculations. In both materials
the crystal symmetries are broken by phonon instabilities, which in turn lead
to changes in the electronic bandstructure also observed in experiment. A total
energy landscape analysis shows no tendency towards a purely electronic
instability and we find that a sizeable lattice distortion is needed to open a
bandgap. We conclude that an excitonic instability is not needed to explain the
phase transition in both Ta$_2$NiSe$_5$ and Ta$_2$NiS$_5$.
