  Transformer models using segment-based processing have been an effective
architecture for simultaneous speech translation. However, such models create a
context mismatch between training and inference environments, hindering
potential translation accuracy. We solve this issue by proposing Shiftable
Context, a simple yet effective scheme to ensure that consistent segment and
context sizes are maintained throughout training and inference, even with the
presence of partially filled segments due to the streaming nature of
simultaneous translation. Shiftable Context is also broadly applicable to
segment-based transformers for streaming tasks. Our experiments on the
English-German, English-French, and English-Spanish language pairs from the
MUST-C dataset demonstrate that when applied to the Augmented Memory
Transformer, a state-of-the-art model for simultaneous speech translation, the
proposed scheme achieves an average increase of 2.09, 1.83, and 1.95 BLEU
scores across each wait-k value for the three language pairs, respectively,
with a minimal impact on computation-aware Average Lagging.
