  Sequence-to-sequence learning naturally has two directions. How to
effectively utilize supervision signals from both directions? Existing
approaches either require two separate models, or a multitask-learned model but
with inferior performance. In this paper, we propose REDER (Reversible Duplex
Transformer), a parameter-efficient model and apply it to machine translation.
Either end of REDER can simultaneously input and output a distinct language.
Thus REDER enables reversible machine translation by simply flipping the input
and output ends. Experiments verify that REDER achieves the first success of
reversible machine translation, which helps outperform its multitask-trained
baselines by up to 1.3 BLEU.
