  We present a study of terahertz frequency magnetoelectric effect in
ferrimagnetic pyroelectric CaBaCo$_4$O$_7$ and its Ni-doped variants. The
terahertz absorption spectrum of these materials consists of spin excitations
and low-frequency infrared-active phonons. We studied the
magnetic-field-induced changes in the terahertz refractive index and absorption
in magnetic fields up to 17 T. We find that the magnetic field modulates the
strength of infrared-active optical phonons near 1.2 and 1.6 THz. We use the
Lorentz model of the dielectric function to analyze the measured magnetic-field
dependence of the refractive index and absorption. We propose that most of the
magnetoelectric effect is contributed by the optical phonons near 1.6 THz and
higher-frequency resonances. Our experimental results can be used to construct
and validate more detailed theoretical descriptions of magnetoelectricity in
CaBaCo$_{4-x}$Ni$_x$O$_7$.
