  This paper presents a study of the structure dynamics in BaFe$_2$Se$_3$. We
combined first-principle calculations, infrared measurements and a thorough
symmetry analysis. Our study confirms that $Pnma$ cannot be the space group of
BaFe$_2$Se$_3$, even at room temperature. The phonons assignment requires $Pm$
to be the BaFe$_2$Se$_3$ space group, not only in the magnetic phase, but also
in the paramagnetic phase at room temperature. This is due to a strong coupling
between a short range spin-order along the ladders, and the lattice degrees of
freedom associated with the Fe-Fe bond length. This coupling induces a change
in the bond-length pattern from an alternated trapezoidal one (as in $Pnma$) to
an alternated small/large rectangular one. Out of the two patterns, only the
latter is fully compatible with the observed block-type magnetic structure.
Finally, we propose a complete symmetry analysis of the BaFe$_2$Se$_3$ phase
diagram in the 0-600\,K range.
