  Hexagonal YMnO$_3$ is well known for the co-occurrence of ferroelectricity
and antiferromagnetism at low temperatures. Using temperature-dependent
spectroscopic ellipsometry at an $a$-plane oriented single crystal, we show how
the dielectric function is affected by the magnetic order transition at the
N\'eel temperature. We focus especially on the pronounced charge transfer
transitions around (1.6-1.7)eV which are strongly connected to Mn 3$d$
electrons. If described with a Bose-Einstein model, the temperature dependency
of their energy and broadening is characterized by effective phonon energies
not larger than 8meV. We argue that this is a hint for the occurrence of a soft
phonon mode related to the antiferromagnetic phase transition. This is observed
in both tensor components of the dielectric function, parallel and
perpendicular to the crystallographic $c$-axis. Furthermore, a suitable
parametrization for the uniaxial dielectric function is presented for the
NIR-VUV spectral range. The broad transitions at energies higher than a
critical point-like bandgap do not show a clear temperature dependence. We also
observe some weak discrete absorption features around the strong charge
transfer transitions with energies matching well to low-temperature
photoluminescence signals.
