  Orbital degrees of freedom mediating an interaction between spin and lattice
were predicted to raise strong magnetoelectric effect, i.e. realize an
efficient coupling between magnetic and ferroelectric orders. However, the
effect of orbital fluctuations have been considered only in a few
magnetoelectric materials, as orbital degeneracy driven Jahn-Teller effect
rarely couples to polarization. Here, we explore the spin-lattice coupling in
multiferroic Swedenborgites with mixed valence and Jahn-Teller active
transition metal ions on a stacked triangular/Kagome lattice using infrared and
dielectric spectroscopy. On one hand, in CaBa$M_4$O$_7$ ($M$ = Co, Fe), we
observe strong magnetic order induced shift in the phonon frequencies and a
corresponding large change in the dielectric response. Remarkably, as an
unusual manifestation of the spin-phonon coupling, the spin-fluctuations reduce
the phonon life-time by an order of magnitude at the magnetic phase
transitions. On the other hand, lattice vibrations, dielectric response, and
electric polarization show no variation at the N\'eel temperature of
CaBaFe$_2$Co$_2$O$_7$, which is built up by orbital singlet ions. Our results
provide a showcase for orbital degrees of freedom enhanced magnetoelectric
coupling via the example of Swedenborgites.
