  We report on the structural and spectroscopic characterization of the
multiferroic Fe$_2$Mo$_3$O$_8$. Synchrotron x-ray and neutron diffraction, as
well as thermal expansion measurements reveal a lattice anomaly at
$T_{\mathrm{N}}\simeq 60\,$K but do not show any symmetry lowering in the
magnetically ordered state. The lattice parameter $c$ exhibits a non-monotonic
behavior with a pronounced minimum around $200\,$K, which is also reflected in
an anomalous behavior of some of the observed infrared-active optical
excitations and parallels the onset of short-range magnetic order. The infrared
reflectivity spectra measured between 5 and 300$\,$K in the frequency range of
$100-8000\,$cm$^{-1}$ reveal most of the expected phonon modes in comparison
with the eigenfrequencies obtained by density-functional calculations. The
$A_1$ phonons show an overall hardening upon cooling, whereas a non-monotonic
behavior is observed for some of the $E_1$ modes. These modes also show a
strongly increased phonon lifetime below $T_\mathrm{N}$, which we associate
with the quenched direction of the orbital moment in the magnetically ordered
state. A similar increase is observed in the lifetime of the higher-lying
$d$-$d$ excitations of the tetrahedral Fe$^{2+}$ site, which become clearly
visible below $T_\mathrm{N}$ only.
