  We present a combined study from angle-resolved photoemission and
density-functional theory calculations of the temperature-dependent electronic
structure in the excitonic insulator candidate Ta$_2$NiSe$_5$. Our experimental
measurements unambiguously establish the normal state as a semimetal with a
significant band overlap of $>$100~meV. Our temperature-dependent measurements
indicate how these low-energy states hybridise when cooling through the
well-known 327~K phase transition in this system. From our calculations and
polarisation-dependent photoemission measurements, we demonstrate the
importance of a loss of mirror symmetry in enabling the band hybridisation,
driven by a shear-like structural distortion which reduces the crystal symmetry
from orthorhombic to monoclinic. Our results thus point to the key role of the
lattice distortion in enabling the phase transition of Ta$_2$NiSe$_5$.
