  When translating words referring to the speaker, speech translation (ST)
systems should not resort to default masculine generics nor rely on potentially
misleading vocal traits. Rather, they should assign gender according to the
speakers' preference. The existing solutions to do so, though effective, are
hardly feasible in practice as they involve dedicated model re-training on
gender-labeled ST data. To overcome these limitations, we propose the first
inference-time solution to control speaker-related gender inflections in ST.
Our approach partially replaces the (biased) internal language model (LM)
implicitly learned by the ST decoder with gender-specific external LMs.
Experiments on en->es/fr/it show that our solution outperforms the base models
and the best training-time mitigation strategy by up to 31.0 and 1.6 points in
gender accuracy, respectively, for feminine forms. The gains are even larger
(up to 32.0 and 3.4) in the challenging condition where speakers' vocal traits
conflict with their gender.
