Improving Zero-shot Neural Machine Translation \\on Language-specific Encoders-Decoders % Conference Paper Title*\\ % {\footnotesize \textsuperscript{*
\IEEEauthorblockN{Junwei Liao
Recently, universal neural machine translation (NMT) with shared encoder-decoder gained good performance on zero-shot translation. Unlike universal NMT, jointly trained language-specific encoders-decoders aim to achieve universal representation across non-shared modules, each of which is for a language or language family. The non-shared architecture has the advantage of mitigating internal language competition, especially when the shared vocabulary and model parameters are restricted in their size. However, the performance of using multiple encoders and decoders on zero-shot translation still lags behind universal NMT. In this work, we study zero-shot translation using language-specific encoders-decoders. We propose to generalize the non-shared architecture and universal NMT by differentiating the Transformer layers between language-specific and interlingua. By selectively sharing parameters and applying cross-attentions, we explore maximizing the representation universality and realizing the best alignment of language-agnostic information. We also introduce a denoising auto-encoding (DAE) objective to jointly train the model with the translation task in a multi-task manner. Experiments on two public multilingual parallel datasets show that our proposed model achieves a competitive or better results than universal NMT and strong pivot baseline. Moreover, we experiment incrementally adding new language to the trained model by only updating the new model parameters. With this little effort, the zero-shot translation between this newly added language and existing languages achieves a comparable result with the model trained jointly from scratch on all languages.
conference_101719.tex
\documentclass[conference]{IEEEtran}
\IEEEoverridecommandlockouts
% The preceding line is only needed to identify funding in the first footnote. If that is unneeded, please comment it out.
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{xcolor}
\def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
    T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}

% mathematics sign and equation
\usepackage{amsmath}
\usepackage{bm}

% algorithm
\usepackage[linesnumbered,ruled,vlined]{algorithm2e}
\SetKwInOut{Input}{Input}

% table
% \usepackage{graphicx}  % resizebox
\usepackage{booktabs}
\usepackage{multirow}

% graphic
\usepackage{subfig}

% disallowed package
% \usepackage{color} 

\usepackage{cleveref}
\crefname{section}{\S}{\S\S}
% \Crefname{section}{\S}{\S\S}
\crefname{table}{Table}{}
\crefname{figure}{Fig.}{}
\crefname{algorithm}{Algorithm}{}
\crefname{equation}{}{}
\crefname{appendix}{Appendix}{}
\crefformat{section}{\S#2#1#3}  %

% Language tags
\newcommand{\De}{\ensuremath{\mathrm{\texttt{De}}}}
\newcommand{\En}{\ensuremath{\mathrm{\texttt{En}}}}
\newcommand{\Es}{\ensuremath{\mathrm{\texttt{Es}}}}
\newcommand{\Fr}{\ensuremath{\mathrm{\texttt{Fr}}}}
\newcommand{\Ar}{\ensuremath{\mathrm{\texttt{Ar}}}}
\newcommand{\Ru}{\ensuremath{\mathrm{\texttt{Ru}}}}
\newcommand{\Zh}{\ensuremath{\mathrm{\texttt{Zh}}}}

\begin{document}

\title{Improving Zero-shot Neural Machine Translation \\on Language-specific Encoders-Decoders
% Conference Paper Title*\\
% {\footnotesize \textsuperscript{*}Note: Sub-titles are not captured in Xplore and
% should not be used}
% \thanks{Identify applicable funding agency here. If none, delete this.}
}

\author{\IEEEauthorblockN{Junwei Liao}
\IEEEauthorblockA{\textit{Department of Computer Science} \\
\textit{University of Electronic Science and Technology of China}\\
Chengdu, China \\
junwei.liao@outlook.com}
\and
\IEEEauthorblockN{Yu Shi}
\IEEEauthorblockA{\textit{Cognitive Services Research Group} \\
\textit{Microsoft}\\
Seattle, USA \\
yushi@microsoft.com}
\and
\IEEEauthorblockN{Ming Gong}
\IEEEauthorblockA{\textit{STCA NLP Group} \\
\textit{Microsoft}\\
Beijing, China \\
migon@microsoft.com}
\and
\IEEEauthorblockN{Linjun Shou}
\IEEEauthorblockA{\textit{STCA NLP Group} \\
\textit{Microsoft}\\
Beijing, China \\
lisho@microsoft.com}
\and
\IEEEauthorblockN{Hong Qu}
\IEEEauthorblockA{\textit{Department of Computer Science} \\
\textit{University of Electronic Science and Technology of China}\\
Chengdu, China \\
hongqu@uestc.edu.cn}
\and
\IEEEauthorblockN{Michael Zeng}
\IEEEauthorblockA{\textit{Cognitive Services Research Group} \\
\textit{Microsoft}\\
Seattle, USA \\
nzeng@microsoft.com}
}

\maketitle

\begin{abstract}
Recently, universal neural machine translation (NMT) with shared encoder-decoder gained good performance on zero-shot translation. Unlike universal NMT, jointly trained language-specific encoders-decoders aim to achieve universal representation across non-shared modules, each of which is for a language or language family. The non-shared architecture has the advantage of mitigating internal language competition, especially when the shared vocabulary and model parameters are restricted in their size. 
However, the performance of using multiple encoders and decoders on zero-shot translation still lags behind universal NMT.
In this work, we study zero-shot translation using language-specific encoders-decoders. We propose to generalize the non-shared architecture and universal NMT by differentiating the Transformer layers between language-specific and interlingua. By selectively sharing parameters and applying cross-attentions, we explore maximizing the representation universality and realizing the best alignment of language-agnostic information. We also introduce a denoising auto-encoding (DAE) objective to jointly train the model with the translation task in a multi-task manner.
Experiments on two public multilingual parallel datasets show that our proposed model achieves a competitive or better results than universal NMT and strong pivot baseline. Moreover, we experiment incrementally adding new language to the trained model by only updating the new model parameters. With this little effort, the zero-shot translation between this newly added language and existing languages achieves a comparable result with the model trained jointly from scratch on all languages.
\end{abstract}

\begin{IEEEkeywords}
multilingual neural machine translation, zero-shot, denoising auto-encoding, language-specific encoders-decoders
\end{IEEEkeywords}

\section{Introduction}

Universal neural machine translation (NMT) draws much attention from the machine translation community in recent years, especially for zero-shot translation, which is first demonstrated by \cite{johnson2017google}. 
Later on, many subsequent works are proposed to further improve the zero-shot performance of universal NMT \cite{gu2019improved,ji2020cross,al2019consistency,arivazhagan2019missing,sestorain2018zero,he2016dual}. However, there are several shortcomings with this shared architecture (\cref{fig:universal}): (1) Since all languages share the same vocabulary and weights, the entire model needs to be retrained from scratch when adapting to a new group of languages; (2) The shared vocabulary grows dramatically when adding many languages, especially for those who do not share the alphabet, such as English and Chinese. The unnecessarily huge vocabulary makes the computation costly in decoding and hinders the deployment of commercial products; (3) This structure can only take text as input and cannot directly add raw signals from other modalities such as image or speech.


\begin{figure}[tb]
\centering
\subfloat[][Universal encoder-decoder]{
\includegraphics[width=0.47\columnwidth]{figures/universal_encoder-decoder.pdf}
\label{fig:universal}}
\subfloat[][Language-specific encoders-decoders]{
\includegraphics[width=0.47\columnwidth]{figures/language-specific_encoders-decoders.pdf}
\label{fig:language-specific}}
\caption{Two typical architectures of multilingual NMT.}
\label{fig1}
\end{figure}

% \begin{figure*}[t]
% \centering
% \subfloat[][Non-contextual word embeddings alignment]{
% \includegraphics[width=1\columnwidth]{content/align-word-single.pdf}
% \label{fig:align-noncontextual-word}}
% \subfloat[][Contextual word embedding alignment]{
% \includegraphics[width=1\columnwidth]{content/align-word-ctx.pdf}
% \label{fig:align-contextual-word}}
% \caption{Alignment of word-level representations from monolingual BERT models on subset of MUSE benchmark. \cref{fig:align-noncontextual-word} and \cref{fig:align-contextual-word} are not comparable due to different embedding vocabularies.}\label{fig:align-word}
% \end{figure*}

Another research direction in machine translation is the jointly trained language-specific encoders-decoders \cite{firat2016multi, lu2018neural, vazquez2019multilingual, escolano2019bilingual, escolano2020multilingual, escolano2020training}, as shown in \cref{fig:language-specific}. Due to the unshared architecture, these works' main goal is to improve the multilingual translation rather than zero-shot transfer. Hence the zero-shot performance of language-specific encoders-decoders still lags behind the universal model.
To solve this problem, previous works add extra network layers as ``interlingua" that is shared by all encoders and decoders \cite{firat2016multi, lu2018neural, vazquez2019multilingual}. 

In this paper, we focus on improving the zero-shot translation on language-specific encoders-decoders. We propose a novel interlingua mechanism that leverages the advantages of both architectures. Specifically, we fully exploit the characteristics of Transformer architecture \cite{vaswani2017attention} without introducing extra network layers. By selectively sharing parameters and applying cross-attentions, we explore maximizing the representation universality and realizing the best alignment of language-agnostic information. Our method can provide good enough universal representation for zero-shot translation, making explicit representation alignment not necessary.

Previous works have proved that monolingual data is helpful to zero-shot translation \cite{lu2018neural, vazquez2019multilingual, zhu2020language}, and reconstruction objective is usually used. However, we found that jointly train the model using a denoising auto-encoding (DAE) task together with the translation task is much more beneficial to zero-shot transfer. We argue that denoising objective can exploit monolingual data more efficiently than reconstruction objective which brings a spurious correlation between source and target sentences.

We verify our method on two public multilingual datasets. Europarl has four languages (En, De, Fr, Es) from close related languages. MultiUN has a group of distant languages (En, Ar, Ru, Zh). The results show that our approach achieve competitive or better results than pivot-based baseline and universal architecture on zero-shot translation. Furthermore, we show that our model can add new languages using incremental training without retraining the existing modules.


Our main contributions can be summarized as:
\begin{itemize}
\item We focus on improving zero-shot translation of language-specific encoders-decoders. It keeps the advantage of adding new languages without retraining the system and achieves comparable or better performance than universal encoder-decoder counterpart and pivot-based methods at the same time.
\item We propose a novel interlingua mechanism within the Transformer architecture without introducing extra network layers. We also propose a multi-task training of DAE and translation to exploit monolingual data more efficiently. These methods bring a significant improvement in the zero-shot performance of language-specific encoders-decoders.
\item We empirically explore several important aspects of proposed methods and give a detailed analysis.
\end{itemize}


\section{Related Works}

\subsection{Zero-shot Neural Machine Translation}
Zero-shot NMT has received more interest in recent years increasingly. \cite{gu2019improved} used decoder pretraining and back-translation to ignore spurious correlations in zero-shot translation; \cite{ji2020cross} proposed a cross-lingual pretraining on encoder before training the whole model with parallel data; \cite{al2019consistency} introduced a consistent agreement-based training method that encourages the model to produce equivalent translations of parallel sentences in auxiliary languages; \cite{arivazhagan2019missing} exploited an explicit alignment loss to align the sentence representations of different languages with same means in the high-level latent space; \cite{sestorain2018zero} proposed a fine-tuning technique that uses dual learning \cite{he2016dual}.
% where a language model is used to provide a signal for fine-tuning zero-shot directions. 
These works all use universal encoder-decoder, while our approaches adopt language-specific encoders-decoders.


\subsection{Language-specific Encoders-Decoders}
% Multilingual NMT can refer to translating from one-to-many languages \cite{dong2015multi}, from many-to-one \cite{zoph2016multi} and many-to-many \cite{johnson2017google}. Within the many-to-many paradigm, existing approaches can be further divided into universal or language-specific encoder-decoders, which is the approach that we follow. 
Most of the works on multilingual NMT use universal encoder-decoder.
% for not requiring modifications of a standard neural machine translation model and transfer learning ability via common parameters shared among the different languages. But this architecture also limit its extensibility to new languages or modalities. 
Few works studied the language-specific encoders-decoders that is more flexible than universal encoder-decoder. 
\cite{firat2016multi} proposed extending the bilingual recurrent NMT architecture to the multilingual case by designing a shared attention-based mechanism between the language-specific encoders and decoders. % to create a language-independent representation. 
\cite{lu2018neural} introduced a neural interlingua into language-specific encoders-decoders NMT architecture that captures language-independent semantic information in its sentence representation. \cite{vazquez2019multilingual} incorporated a self-attention layer, shared among all language pairs, that serves as a neural interlingua. 
% This layer exploits the semantics from each language for performing translation and develops into a language-independent meaning representation that can efficiently be used for transfer learning. 
A series of works \cite{escolano2019bilingual,escolano2020multilingual,escolano2020training} 
adopted different training strategies to improve multilingual NMT performance on language-specific encoders-decoders without any shared parameters. Although these works make some progress on improving the performance of multilingual translation at different levels, they still lag behind universal encoder-decoder on zero-shot translation.


\subsection{Parameter Shared Methods for Multilingual NMT}
Several proposals promote cross-lingual transfer, the key to zero-shot translation, by modifying the model’s architecture and selectively sharing parameters.
% Recent works propose different methods for parameter sharing between language pairs in multilingual NMT.
\cite{blackwood2018multilingual} proposed sharing all parameters but the attention mechanism. % and show improvements over sharing all parameters. 
\cite{zaremoodi2018adaptive} utilized recurrent units with multiple blocks together with a trainable routing network.
\cite{platanios2018contextual} develop a contextual parameter generator that can be used to generate the encoder-decoder parameters for any source-target language pair. 
% \citet{gu2018universal} proposed a ``Universal Language Representation'' layer together with a Mixture-of-Language-Experts component to improve a many-to-one model from 5 languages into English.
Our work uses parameter sharing based on Transformer \cite{vaswani2017attention}. The most similar work like ours is \cite{sachan2018parameter} but with some major differences:
% that explores sharing various components in Transformer models. But there are some major differences with us: 
(1)They study the parameter sharing methods only on one-to-many translation while we focus on zero-shot translation in many-to-many scenario; (2) They share the partial self-attention weights in all decoders layers while we share the selective layers of encoders; (3) They use one shared vocabulary while we use separated vocabulary for each language, which is more flexible to add new languages without retraining the whole system. 
% Emerging Cross-lingual Structure in Pretrained Language Models: share layers of encoder but they study cross-lingual transfer for multilingual masked language modeling that only consists of encoder of Transformer.


\section{Approach}

In this section, we first introduce the language-specific encoders-decoders adopted in our method. Then we propose our approach -- combining interlingua via parameter sharing and DAE task jointly trained with translation task -- to improve the zero-shot translation on language-specific encoders-decoders.

\subsection{Background: Language-specific Encoders-Decoders for Multilingual NMT}

Language-specific encoder-decoder for multilingual NMT is based on the sequence-to-sequence model except that, as \cref{fig:language-specific} illustrates, each language has its own encoder and decoder. We denote the encoder and the decoder for the $i_{th}$ language in the system as $enc_i$ and $dec_i$, respectively. For language-specific scenarios, both the encoder and decoder are considered independent modules that can be freely interchanged to work in all translation directions. 
% \YS{Mention "leave language family scenario for future work" somewhere in the paper.}
We use $(x_i, y_j)$ where $i, j \in \{1,...,K\}$ to represent a pair of sentences translating from a source language $i$ to a target language $j$. $K$ languages are considered in total. Our model is trained by maximizing the likelihood over training sets $D_{i,j}$ of all available language pairs $\mathcal{S}$. 
% \YS{We didn't use all language pairs in training, only x-en and en-x.} 
Formally, we aim to maximize $\mathcal{L}_{mt}$: 
\begin{equation}
    \mathcal{L}_{mt}(\theta) = 
    \sum_{\begin{subarray}{c}(x_i, y_j)\in D_{i, j},\\ (i, j) \in \mathcal{S}\end{subarray}}
    \log p(y_j | x_i; \theta),
    \label{eq1}
\end{equation}
where the probability $p(y_j | x_i)$ is modeled as
\begin{equation}
    p(y_j | x_i) = dec_j(enc_i(x_i)).
    \label{eq2}
\end{equation}
% \YS{Does any other paper use equation as Equ (2)? To me, decoder output is a sequence but a probability.}

\cite{johnson2017google} showed that a trained multilingual NMT system could automatically translate between unseen pairs without any direct supervision, as long as both source and target languages were included in training. In other words, a model trained, for instance, on Spanish $\rightarrow$ English and English $\rightarrow$ French can directly translate from Spanish to French. Such an emergent property of a multilingual system is called zero-shot translation. It is conjectured that zero-shot NMT is possible because the optimization encourages different languages to be encoded into a shared space so that the decoder is detached from the source languages.
Universal encoder-decoder can naturally possess this property because all language pairs share the same encoder and decoder. But for language-specific encoders-decoders, there is no shared part among languages. That makes transfer learning hardly take effect to transfer the knowledge learned in high-source language to low-source language. That explains why language-specific encoders-decoders underperform universal encoder-decoder on zero-shot translation. \cite{escolano2020multilingual} also attributes this problem to limited shared information. To improve the zero-shot translation on language-specific encoders-decoders, we propose the interlingua via parameter sharing and DAE task.


% need natural transition from up paragraph to down paragraph. don't make reviewer to think that previous work about language-specific model have study the zero-shot. It's contradict with our claim.

\subsection{Interlingua via Parameter Sharing and Selective Cross-attention}
Previous works \cite{firat2016multi,lu2018neural,vazquez2019multilingual} propose to use an interlingua layer to bring the shared space to language-specific encodes-decoders. Interlingua is a shared component between encoders and decoders, which maps the output of language-specific encoders into a common space and gets an intermediate universal representation as the input to decoders. They implemented interlingua as extra network layers that make the model complicated and training inefficient. Unlike their methods, we propose implementing an interlingua by sharing the Transformer \cite{vaswani2017attention} layers of language-specific encoders. \cref{fig:interlingua} gives a schematic diagram for our proposed model.

\begin{figure}[tb]
\centering
\includegraphics[width=1\columnwidth]{figures/encoders_interlingua_decoders.pdf} 
\caption{Interlingua via parameter sharing and selective cross-attention. We use English and French as two example languages for illustrating.}
\label{fig:interlingua}
\end{figure}

\cite{conneau2020emerging,artetxe2020cross} show that transfer is possible even when there is no shared vocabulary across the monolingual corpora. The only requirement is that there are some shared parameters in the top layers of the multilingual encoder. Inspired by their work, we share the top layers of language-specific encoders while keeping the embedding and low layers intact. In this way, the model can get the transfer learning ability via parameter sharing while maintaining the multi-encoders' flexibility via independent vocabulary.
In the language-specific decoders' side, we keep the decoders separately without sharing parameters. 
% \YS{I'd encourage to add experiments and analyses that can support this decision.}

Intuitively, to generate zero-shot language sentences from interlingua representation in a common shared space, cross-attention between interlingua and decoder layers should capture the high-level information, such as semantics, which is language-agnostic. \cite{tan2019study} shows that for multilingual NMT, the top layers of the decoder capture more language-specific information. Base on their conclusion, we can infer that low layers of decoder capture more language-agnostic information. Combining the two points above, we only conduct cross-attention between the interlingua and low layers of the decoder to capture language-agnostic knowledge more effectively.
We denote the $enc'_i$ as the language-specific part of encoder $i$, $dec'_j$ as the decoder $j$ with modified cross-attention. Equation \cref{eq2} is updated to
\begin{equation}
    p(y_j | x_i) = dec'_j(interlingua(enc'_i(x_i))),
    \label{eq3}
\end{equation}
where $interlingua$ denotes the interlingua via shared Transformer layers. 
% \YS{Same question with equation (2).}


\subsection{Denoising Auto-encoder (DAE) Task}
% Previous works have proved that monolingual data is useful to improve performance of zero-shot translation for language-specific encoders-decoders \cite{lu2018neural,vazquez2019multilingual,escolano2019bilingual}. 
% Explaination: Since WMT15 is not a multi-parallel corpus, the model essentially learns to handle two separate tasks, namely translation from English and translation to English. For instance, since the output of the De encoder and the En encoder would never be used by the same decoder, there is no reason for De and En source sentences to share the same embedding, even if they are translations of each other.
% They incorporate an reconstruction loss for the purpose of using monolingual data. 
% We denote the $x_i'=dec_i(interlingua(enc_i(x_i)))$. The reconstruction loss is computed crossentropy between $x_i'$ and $x_i$. 
% Inspired by \citet{liu2020multilingual}, which shows that pretraining a multilingual sequence-to-sequence denoising auto-encoder can efficiently improve the downstream task performance, we propose to use denoising auto-encoder objective to replace the reconstruction objective. 
We propose to use a DAE task and train with translation task jointly.
Our training data cover $\mathcal{K}$ languages, and each $D_k$ is a collection of monolingual sentences in language $k$. We assume access to a noising function $g$, defined below, that corrupts text, and train the model to predict the original text $x_k$ given $g(x_k)$. More formally, we aim to maximize $\mathcal{L}_{dae}$ as
\begin{equation}
    \mathcal{L}_{dae}(\theta) = 
    \sum_{\begin{subarray}{c}x_k \in D_k,\\ k \in \mathcal{K}\end{subarray}}
    \log p(x_k | g(x_k); \theta),
    \label{eq4}
\end{equation}
where $x_k$ is an instance in language $k$ and the probability $p(x_k | g(x_k))$ is defined by
\begin{equation}
    p(x_k | g(x_k)) = dec'_k(interlingua(enc'_k(g(x_k)))).
    \label{eq5}
\end{equation}

The noising function $g$ injects three types of noise to obtain the randomly perturbed text: First, we randomly drop tokens of the sentence with a probability; Second, we substitute tokens with the special masking token with another probability; Third, the token order is locally shuffled. The candidate tokens can be in a unit of subword or whole word and span to N-grams.

Finally, the objective function of our learning algorithm is
\begin{equation}
    \mathcal{L} = \mathcal{L}_{mt} + \mathcal{L}_{dae}. % + \mathcal{L}_{dist}.
    \label{eq8}
\end{equation}
We jointly train the multiple encoders and decoders by %alternatively 
randomly selecting the two tasks with equal probability, i.e. multilingual translation and DAE task. 


\section{Experiments}
\subsection{Setup}
We evaluate the proposed approaches against several strong baselines on two public multilingual datasets across a variety of language, Europarl\footnote{http://www.statmt.org/europarl/} \cite{koehn2005europarl} and MultiUN\footnote{http://opus.nlpl.eu/MultiUN.php} \cite{eisele2010multiun}. In all experiments, we use BLEU \cite{papineni2002bleu} as the automatic metric for translation evaluation.\footnote{We calculate BLEU scores using toolkit SacreBLEU \cite{post2018call} with default tokenizer except Chinese which uses Zh tokenizer.}

\subsubsection{Datasets}

\begin{table}[tb]
	\caption{Overall dataset statistics where each pair has a similar number of training samples. All the remaining directions are used to evaluate zero-shot performance. % \YS{The table is misleading. Are the "parallel pairs" training pairs? What are the dev and test pairs?}
	}
	\centering
	\scalebox{0.9}{
	\begin{tabular}{l | l r }
	\toprule
	Dataset & parallel pairs & size/pair \\
	\midrule
    Europarl   & Es-En, De-En, Fr-En  & 0.6M \\
    \midrule
    MultiUN & Ar-En, Ru-En, Zh-En & 2M \\
    \bottomrule

	\end{tabular}}
	\label{table:data}
	\vspace{0pt}
\end{table}

The detailed statistics of Europarl and MultiUN datasets are in \cref{table:data}. To simulate the zero-shot settings, the training set only allowing parallel sentences from/to English, where English acts as the pivot language. For Europarl corpus, to compare with previous work fairly, we follow \cite{al2019consistency}'s method to preprocess the Europarl to avoid multi-parallel sentences in training data and ensure the zero-shot setting.
% Specifically: To do so, we follow Cheng et al. (2017); Chen et al. (2017) and randomly split the overlapping pivot sentences of the original source-pivot and pivot-target corpora into two parts and merge them separately with the non-overlapping parts for each pair.
We use the \textit{dev2006} as the validation set and the \textit{test2006} as the test set which contain 2,000 multi-parallel sentences. For vocabulary, We use SentencePiece \cite{kudo2018sentencepiece} to encode text as WordPiece tokens \cite{kudo2018subword}. Due to the language-specific architecture of our model, each language has its own vocabulary. We choose a vocabulary of 32K wordpieces for each language. 
% \YS{May not need to mention vocab size tuning.}

For MultiUN corpus, % we choose distant related languages (En, Ar, Ru, Zh). To compare with previous work fairly, 
We randomly sub-sample the 2M sentences per language pair for the training set and 2000 sentences per language pair for validation and test sets. Vocabulary size for each language is also 32K.

For DAE task, we only use monolingual data extracted from parallel training data. No extra monolingual data are used.

\subsubsection{Model}
Our code was implemented using PyTorch on top of the Huggingface Transformers library\footnote{https://github.com/huggingface/Transformers}. % and our language-specific encoders and decoders model is based on T5 model \cite{raffel2020exploring} which use Transformer \cite{vaswani2017attention} architecture.
% T5  is an encoder-decoder model pretrained on a multi-task mixture of unsupervised and supervised tasks and for which each task is converted into a text-to-text format.
% Different with T5, each language has its own encoder and decoder and independent vocabulary. 
To decouple the output representation of an encoder from the task, we use the target language %ID 
code as the initial token of the decoder input following the practice of \cite{gu2019improved}.
The Transformer layers in our model use the parameters of $d_{model}=512, d_{ff}=2048, n_{heads}=8$. The encoders have 9 layers with 6 top layers sharing parameters as interlingua. The decoders have 12 layers with 6 low layers conducting cross-attention with interlingua.
The output softmax layer is tied with input embeddings \cite{press2017using}.


\subsubsection{Training}
We train our models with the RAdam optimizer \cite{liu2019variance} and the inverse square root learning rate scheduler of \cite{vaswani2017attention} with 5e-4 learning rate and 64K linear warmup steps. For each model, we train it using 8 NVIDIA V100 GPUs with 32GB of memory and mixed precision. It takes around three days to train one model. We use batch size 32 for each GPU. We stop training at optimization step 200K and select the best model based on the validation set.
We search the following hyperparameter for zero-shot translation: batch size \{32, 96\}; learning rate \{5e-4, 1e-3, 2e-3\}; linear warmup steps \{4K, 10K, 32K, 40K, 64K\}.
% According to \citet{gu2019improved}, setting dropout rate on each attention head to 0 is helpful to the zero-shot translation and has no effect on supervised translation directions, but we cannot observed that boost in our experiment. Thus 
We use dropout 0.1 throughout the whole model.
% others: label smoothing 0.2, grad clipping 2, MAX_LEN 512, MAX_GEN_LENGTH 256
For decoding, we use beam-search with beam size 1 for all directions.
The noise function uses token deletion rate 0.2, token masking rate 0.1, WordPiece token as the token unit, and unigram as span width.

\subsubsection{Evaluation}
We focus our evaluation mainly on zero-shot performance of the following methods:
\begin{itemize}
    \item \textbf{Univ.}, which stands for directly evaluating a multilingual universal encoder-decoder model after standard training \cite{johnson2017google}.
    \item \textbf{Pivot}, which performs pivot-based translation using a multilingual universal encoder-decoder model (after standard training); often regarded as gold-standard.
    \item \textbf{Ours}, which represents language-specific encoders-decoders with shared interlingua and jointly trained with multilingual translation and DAE tasks.
\end{itemize}

To ensure a fair comparison in terms of model capacity, all the techniques above use the same 
Transformer architecture described above, i.e. 9 layers of encoder and 12 layers of decoder. All other results provided in the tables are as reported in the literature.

\subsection{Main Results}
The supervised and zero-shot translation results on Europarl and MultiUN corpora are shown in \cref{tab:europarl} and \cref{tab:multiun}. 

\begin{table}[tb]
\caption{Zero-shot results on Europarl corpus.
}
\centering
\scriptsize
\def\arraystretch{0.8}
\setlength{\tabcolsep}{8.25pt}
\begin{tabular}{@{}lrr|rr|r@{}}
\toprule
                        & \multicolumn{2}{c|}{Previous work}   & \multicolumn{2}{c|}{Our baselines} \\
\cmidrule[0.5pt]{2-5}
                        & BRLM-SA$^\dagger$       & Agree$^\ddagger$     & Univ. & Pivot     & Ours \\
\cmidrule[0.5pt]{1-6}                                                                        
$\En \rightarrow \De$   & 24.95            & 22.44            & 26.05 & ---             & \textbf{26.21} \\
$\De \rightarrow \En$   & ---              & 29.07            & \textbf{31.83} & ---    & 31.81 \\
$\En \rightarrow \Fr$   & \textbf{35.91}   & 32.55            & 34.40 & ---             & 34.29 \\
$\Fr \rightarrow \En$   & ---              & 33.30            & 35.02 & ---             & \textbf{35.40} \\
$\En \rightarrow \Es$   & 34.92            & 33.80            & \textbf{35.46} & ---    & 34.98 \\
$\Es \rightarrow \En$   & ---              & 34.53            & 36.34 & ---             & \textbf{36.37} \\
\cmidrule[0.5pt]{1-6}                                                                   
Supervised (avg.)       & ---              & 30.95            & 33.18 & ---             & \textbf{33.18} \\
\cmidrule[0.5pt]{1-6}                                                                   
$\De \rightarrow \Fr$   & \textbf{30.66}   & 24.45            & 27.35 & 29.35           & 29.41 \\
$\Fr \rightarrow \De$   & ---              & 19.15            & 20.05 & 25.02           & \textbf{25.28} \\
$\De \rightarrow \Es$   & ---              & 22.45            & 28.28 & 30.10           & \textbf{30.48} \\
$\Es \rightarrow \De$   & ---              & 20.70            & 20.55 & 25.15           & \textbf{25.50} \\
$\Fr \rightarrow \Es$   & 37.02            & 29.91            & 33.50 & 34.75           & \textbf{38.21} \\
$\Es \rightarrow \Fr$   & ---              & 30.94            & 34.24 & 34.90           & \textbf{38.20} \\
\cmidrule[0.5pt]{1-6}                                                                   
Zero-shot (avg.)        & ---              & 24.60            & 27.33 & 29.88           & \textbf{31.18} \\
\bottomrule
\end{tabular}\\
$^\dagger$BRLM-SA \cite{ji2020cross}. $^\ddagger$Agree \cite{al2019consistency}.
% \vspace{-1ex}
\label{tab:europarl}
\end{table}

\begin{table}[tb]
\caption{Zero-shot results on MultiUN corpus.
}
\centering
\scriptsize
\def\arraystretch{0.8}
\setlength{\tabcolsep}{8.25pt}
\begin{tabular}{@{}lr|rr|r@{}}
\toprule
                        & Previous work         & \multicolumn{2}{c|}{Our baselines}  \\
\cmidrule[0.5pt]{2-4}
                        & ZS+LM$^\dagger$       & Univ. & Pivot               & Ours  \\
\cmidrule[0.5pt]{1-5}
$\En \rightarrow \Ar$   & ---                     & 37.07 & ---               & \textbf{38.22} \\
$\Ar \rightarrow \En$   & ---                     & 51.82 & ---               & \textbf{52.82} \\
$\En \rightarrow \Ru$   & ---                     & 40.83 & ---               & \textbf{42.02} \\
$\Ru \rightarrow \En$   & ---                     & 47.52 & ---               & \textbf{48.57} \\
$\En \rightarrow \Zh$   & ---                     & 52.51 & ---               & \textbf{52.64} \\
$\Zh \rightarrow \En$   & ---                     & 47.74 & ---               & \textbf{48.63} \\
\cmidrule[0.5pt]{1-5}                                                   
Supervised (avg.)       & 45.80                   & 46.25 & ---               & \textbf{47.15} \\
\cmidrule[0.5pt]{1-5}                                                           
$\Ar \rightarrow \Ru$   & 21.5                    & 27.96 & \textbf{36.68}    & 35.04 \\
$\Ru \rightarrow \Ar$   & 28.0                    & 25.65 & \textbf{30.95}    & 29.33 \\
$\Ar \rightarrow \Zh$   & \textbf{43.8}           & 37.96 & 43.64             & 41.18 \\
$\Zh \rightarrow \Ar$   & 27.3                    & 22.67 & \textbf{28.46}    & 26.27 \\
$\Ru \rightarrow \Zh$   & \textbf{43.3}           & 31.70 & 36.33             & 34.38 \\
$\Zh \rightarrow \Ru$   & 19.9                    & 21.86 & \textbf{29.05}    & 25.56 \\
\cmidrule[0.5pt]{1-5}                                                  
Zero-shot (avg.)        & 30.6                    & 27.97 & \textbf{34.18}    & 31.96 \\
\bottomrule
\end{tabular}\\
$^\dagger$ZS+LM \cite{gu2019improved}. 
% \vspace{-1ex}
\label{tab:multiun}
\end{table}


\subsubsection{Results on Europarl Dataset} 
For zero-shot translation, our model outperforms all other baselines except on De$\rightarrow$Fr to BRSM-SA \cite{ji2020cross} that use a large scale monolingual data to pretrain encoder. Especially, Our model outperforms the pivot-based translation in all directions. For Fr$\leftrightarrow$Es pairs, our model improves about 3.4 BLEU points over the pivoting. The pivot-based translation is a strong baseline in the zero-shot scenario that often beats the other multilingual NMT system baselines (\cite{johnson2017google,al2019consistency}). 
% Pivot definition: how implicitly-learned bridging (zero-shot translation) performs in comparison to explicit bridging (i.e., first translating to a common language like English and then translating from that common language into the desired target language) as typically used in machine translation systems.
Pivoting translates source to pivot then to target in two steps, causing an inefficient translation process. Our approaches translate source to target directly between any zero-shot directions, which is more efficient than pivoting. 
For supervised translation, our model gets the same averaged score with universal NMT and receives a better score in most direction (En$\rightarrow$De, Fr$\rightarrow$En, Es$\rightarrow$En). Baseline Agree \cite{al2019consistency} using agreement loss manifests the worst performance. We conjecture that it is because their model is based on LSTM \cite{hochreiter1997long}, which was proved inferior to Transformer on multilingual NMT\cite{lakew2018comparison}. But regarding the BRLM-SA \cite{ji2020cross} that uses Transfomer-big and cross-lingual pretraining with large scale monolingual, our model only uses the parallel data and get a better score on most of supervised and zero-shot direction.

\subsubsection{Results on MultiUN Dataset}
Our model performs better for supervised and zero-shot translation than universal NMT and the enhanced universal NMT with language model pretraining for decoder \cite{gu2019improved} that denoted as ZS+LM in \cref{tab:multiun}. But our model still lags behind the pivoting 
%, which uses universal NMT to translate source to pivot then to target in two separate steps with each step receiving supervised signal of parallel corpora, 
about 2.2 BLEU points. Similar results were observed in previous works \cite{gu2019improved,ji2020cross} where their methods based on universal encoder-decoder also underperform pivoting on MultiUN dataset. 
% We conjecture that unlike Europarl, which is preprocessed to remove the overlapped sentences between \textit{source-pivot} and \textit{pivot-target} language pairs, MultiUN include pivot
% For pairs which are lexically more similar compared to the pivot language (e.g. Es-Fr v.s. En), ZS+LM achieved much better performance than its pivot-based counterpart.
To surpass the pivoting, they introduced the back-translation \cite{sennrich2016improving} to generate pseudo parallel sentences for all zero-shot direction based on pretrained models, and further trained their model with these pseudo data. 
% Back-translation is computationally more expensive for needing to create synthetic parallel corpora for all language pairs (up to $O(K^2)$) to train a multilingual model for $K$ languages and the performance of ultimate model might be affected by the quality of the pretrained models. 
Strictly speaking, their methods are ``zero-resource'' translation rather than ``zero-shot'' translation since the model has seen explicit examples of zero-shot language pair during training, which zero-shot translation should not do \cite{johnson2017google}. 
% the difference between “zero-shot” and “zero-resource” translation is the additional fine-tuning step which is required in the latter approach.
Considering this reason, although our model can also benefit from adding training data augmented by back-translation, we decide not to use it to further improve our model just for beating the pivoting baseline. 


\section{Discussion}
In this section, we discuss some important aspects of the proposed approach. % ablation experiment and answer following questions about our approach.

\subsection{Incremental Language Expansion}

%As we mentioned above, language-specific encoders-decoders can solve the drawbacks of universal encoder-decoder on multilingual NMT. One of the main points is the former can add a new language without retraining existing modules. We have improved zero-shot performance of the language-specific encoders-decoders to reach competitive or better results than universal encoder-decoder counterpart. Now we want to know if our approaches can keep the zero-shot capability when adding a new language without retraining a whole model. We use an incremental training \cite{escolano2019bilingual} to add a new language to a trained model. 
First, we explore our model's zero-shot transfer capability when incrementally adding new languages. The incremental training is similar to that in \cite{escolano2019bilingual}. 
For illustration, let us assume we have gotten a language-specific encoders-decoders model with initial training on En$\leftrightarrow$Es,Fr parallel data. Now we want to add a new language, say De, into the existing modules via incremental training only on En$\leftrightarrow$De parallel data. 
% Firstly, we can set up a new bilingual system with language De as source and language En as target. To ensure that the representation produced by this new pair is compatible with the previously jointly trained system, we use the previous En decoder as the decoder of the new De$\rightarrow$En system and freeze it. During training, we optimize the cross-entropy between the generated tokens and En reference data but update only the parameters of the De encoder. By doing so, we train De encoder not only to produce good quality translations but also to produce similar representations to the already trained languages. Following the same principles, the De decoder can also be trained as a bilingual system by freezing the En encoder and training the decoder of the En$\rightarrow$De system by optimizing the cross-entropy with the De reference data. We also jointly train De$\rightarrow$De with proposed noising auto-encoder task. 
% \YS{The description gives people the feeling that we use 3 steps for the incremental training. Why not just simplify this by saying "add De encoder and decoder to the model, randomly init them, freeze which existing encoders and decoders (is English encoder and decoder frozen or not?), what tasks are used (I guess they are en$<->$de translation and dae of en-en and de-de)."}
%We add an encoder-decoder of De into trained model, freeze all existing modules and train De encoder and decoder alternatively with translation task and denoising auto-encoder task on En$\leftrightarrow$De data.
In the initial training, we jointly train three encoders and three decoders for languages En, Es, and Fr, respectively. 
In the incremental training step, we add both a new encoder and a new decoder for De into the initial model, share the interlingua layers with the initial modules, and randomly initialize the non-shared layers. The weight update is only conducted on the newly added non-shared layers using the En$\leftrightarrow$De parallel data by freezing the weights of the six initial modules. Both steps use the proposed multi-task training of DAE and translation objectives.

%Following the same process, we experimented on adding De (Fr or Es) to a trained model with initial training on En$\leftrightarrow$Es,Fr (En$\leftrightarrow$De,Es or En$\leftrightarrow$De,Fr), where the parallel data come from Europarl dataset. 
Following the same process, we also experimented with adding Fr or Es to the model initially trained on other three languages, respectively. 
% To see the effect of incremental training on adding a distant related language to existing model, we also experimented on adding Zh to a trained model with initial training on En$\leftrightarrow$Ar,Ru, where the parallel data come from MultiUN dataset. 
We use the same experimental setting as in the main experiment unless stated otherwise. We compare the zero-shot translation results of the incremental training with the joint training on the Europarl dataset (En$\leftrightarrow$De,Fr,ES). % and MultiUN (En$\leftrightarrow$AR,Ru,Zh).
The results are shown in \cref{tab:add-new-language}.

\begin{table}[tb]
\caption{The results of zero-shot translation on Europarl datasets when adding a new language. The metric is BLEU. \textbf{Joint}, \textbf{Init.}, \textbf{Incr.} represent joint training, initial training, and incremental training, respectively.} 
\centering
\footnotesize
\resizebox{1\linewidth}{!}{
\begin{tabular}[b]{l cc cc cc cc}
\toprule
Europarl & \multicolumn{8}{c}{De, Es, Fr $\leftrightarrow$ En} \\
\midrule
\multirow{2}{*}{Direction} & \multicolumn{2}{c}{Fr-De} & \multicolumn{2}{c}{Es-De} & \multicolumn{2}{c}{Es-Fr} & Zero & Parallel \\
& $\leftarrow$ & $\rightarrow$ & $\leftarrow$ & $\rightarrow$ & $\leftarrow$ & $\rightarrow$ & Avg & Avg \\
\midrule
\textbf{Joint} & 29.41 & 25.28 & 30.48 & 25.50 & 38.21 & 38.20 & \textbf{31.18} & \textbf{33.18} \\
\midrule
\multicolumn{9}{l}{\textit{Intial training on En$\leftrightarrow$Es,Fr, then incremental training on En$\leftrightarrow$De.}} \\
\midrule
\textbf{Init.} &   --- &   --- &   --- &   --- & 37.99 & 37.40 & ---   & ---   \\
\textbf{Incr.} & 28.95 & 24.86 & 30.19 & 25.67 & 37.99 & 37.40 & 30.84 & 32.98 \\
\midrule
\multicolumn{9}{l}{\textit{Intial training on En$\leftrightarrow$De,Es, then incremental training on En$\leftrightarrow$Fr.}} \\
\midrule
\textbf{Init.} &   --- &   --- & 30.58 & 25.18 &   --- &   --- &   --- &   --- \\
\textbf{Incr.} & 29.61 & 24.58 & 30.58 & 25.18 & 37.57 & 37.84 & 30.89 & 32.78 \\
\midrule
\multicolumn{9}{l}{\textit{Intial training on En$\leftrightarrow$De,Fr, then incremental training on  En$\leftrightarrow$Es.}} \\
\midrule
\textbf{Init.} & 29.30 & 24.93 &   --- &   --- &   --- &   --- &   --- &   --- \\
\textbf{Incr.} & 29.30 & 24.93 & 30.60 & 24.82 & 38.24 & 38.03 & 30.99 & 32.83 \\
\bottomrule
\end{tabular}
}
\label{tab:add-new-language}
\end{table}

%In \cref{tab:add-new-language}, we can see that although the joint training gets the better performance on supervised and zero-shot translation, adding a new language through incremental training obtains fairly good results, too. % For Europarl, 
%Adding a different new language only make the average BLEU score drop 0.2$\sim$0.4 point on supervised translation and 0.27$\sim$0.34 point on  zero-shot translation. 
% For MultiUN, adding Zh make the average BLEU score drop 1.85 and 1.43 on supervised and zero-shot translation, respectively. 
%Comparing the initial training and increment training in each group, we can find that adding a new language does not affect the performance of a trained model, which is the advantage what we want to obtain by incremental training.
%Note that in the phase of incremental training, due to our shared interlingua implementation, we only train embedding layer and low layers of Transformer encoder, which further reduces the number of  trainable parameters substantially. In contrast with joint training, which took three days to get the final results, incremental training only took half a day to get the comparable results.
\cref{tab:add-new-language} shows that the universal representation learned in the initial model is easily transferred to the new language. The gap between the incremental training and the joint training is within 0.2$\sim$0.4 points on supervised translation and 0.27$\sim$0.34 points on zero-shot translation. We suspect that the hyperparameters for joint training may not be the best for incremental training. In addition, incremental training followed by continued joint training could further improve the performance. We leave those experiments in future work. 

Besides the good zero-shot transferability, incremental training is also lightweight. Due to the interlingua mechanism, we only update the model weights of the non-interlingua layers of the newly added modules. This dramatically reduces the number of trainable parameters. In contrast with the joint training that takes three days, incremental training only takes half a day to get a comparable result.


\subsubsection{Conclusion}
%Through the analysis in this section, we show that proposed approach can keep the performance of supervised and zero-shot translation without requiring to retrain existing model when adding a new language. We believe that our approach can be particularly useful for situations in which a rapid extension of an existing machine translation system is critical.
The proposed incremental training approach is a lightweight language expansion method that can fully leverage the initial model's interlingua layers and the universal representation learned from the initial training for both parallel and zero-shot languages.

\subsection{Interlingua Structure Analysis}
% To empirically examine the utility of this approach, we examine the case of translation from a common source language to multiple target languages

%In appoarches section, we propose to share some Transformer top layers of language-specific encoders as interlingua and choose some low layers of decoders to conduct cross-attention with interlingua. A natural question is how many layers of encoders should be selected to share parameters and how many layers of decoders should be chosen to carry on cross-attention. We also want to know whether sharing the layers of decoder can benefit the zero-shot translation. So we empirically examine following questions: 1) Does the proposed methods of sharing encoders layers works? 2) If it works, how many encoders layers should we share? 3) How many decoder layers should we use to conduct cross-attention with interlingua? 4) Can we share the decoders layers, too?
In this section, we empirically examine the following questions: 1) How important it is to share the interlingua layers of the encoders? 2) How to partition the language-specific and interlingua layers in the encoders given a fixed total number of layers? 3) Which layers in the decoders should be chosen to conduct cross-attention to the encoder output? 4) Whether sharing interlingua layers among the decoders can benefit the zero-shot translation?

Before the discussion, we first define some shorthands. We denote the encoder interlingua layers between layer $i$ and layer $j$ as \textbf{E$i$-$j$}, the decoder interlingua layers as \textbf{D$i$-$j$}, and the decoder layers that have cross-attention to interlingua output as \textbf{C$i$-$j$}. Both $i$ and $j$ are inclusive and 1-based.

We use the original Europarl corpus without preprocessing. 
% \YS{Does this mean the overlap dataset?} 
To accelerate the experiment, we use a small version of the proposed model. It has a similar architecture to that in our main experiment except that the language-specific encoders have 8 layers and language-specific decoders have 10 layers. 
% We evaluate the models on 100 sentences sampled from the validation set to reduce the evaluation time. 
All the experiments use the same random seeds to ensure the results are comparable.

We design four groups of experiments to answer the questions. Because there can be exponentially many combinations considering all the different feasible sets of configuration, we intuitively only select a subset of these combinations.
The results are reported in \cref{tab:interlingua}.


\begin{table}[tb]
\caption{Dissecting the interlingua structure based on zero-shot performance. The best score in each group is bold. \textbf{E}/\textbf{D}/\textbf{C} represent encoder/decoder/cross-attention. The number after the letter denotes the range of shared encoder layers/shared decoder layers/cross-attention layers respectively.} 
\centering
\footnotesize
\resizebox{1\linewidth}{!}{
\begin{tabular}[b]{l cc cc cc cc}
\toprule
Europarl & \multicolumn{8}{c}{De, Es, Fr $\leftrightarrow$ En} \\
\midrule
\multirow{2}{*}{Direction} & \multicolumn{2}{c}{Fr-De} & \multicolumn{2}{c}{Es-De} & \multicolumn{2}{c}{Es-Fr} & Zero & Parallel \\
& $\leftarrow$ & $\rightarrow$ & $\leftarrow$ & $\rightarrow$ & $\leftarrow$ & $\rightarrow$ & Avg & Avg \\
\midrule
\multicolumn{9}{l}{\textit{1. Ablation of sharing encoders layers.}} \\
\midrule
\textbf{E3-8, C1-10} & 32.1 & 26.8 & 30.8 & 25.3 & 36.8 & 37.4 & \textbf{31.5} & \textbf{31.9} \\
\textbf{C1-10} & 15.5 & 8.6 & 12.4 & 9.3 & 15.1 & 19.1 & 13.3 & 31.6 \\
% \textbf{E3-8, C3-8}$^\ddagger$ & 30.1 & 23.9 & 30.6 & 25.2 & 35.0 & 34.5 & 29.9 & 31.5 \\
\midrule
\multicolumn{9}{l}{\textit{2. Change number of encoders shared layers.}} \\
\midrule
\textbf{E1-8, DC3-8} & 29.3 & 24.2 & 30.1 & 24.6 & 36.2 & 35.9 & \textbf{30.1} & 31.4 \\
\textbf{E3-8, DC3-8} & 30.1 & 23.9 & 30.6 & 25.2 & 35.0 & 34.5 & 29.9 & 31.5 \\
\textbf{E5-8, DC3-8} & 28.0 & 23.8 & 29.8 & 23.2 & 34.9 & 33.8 & 28.9 & 31.2 \\
\textbf{E7-8, DC3-8} & 27.3 & 22.1 & 28.2 & 21.3 & 33.7 & 31.4 & 27.3 & 31.7 \\
\textbf{DC3-8}       & 25.1 & 20.6 & 27.8 & 20.8 & 32.1 & 30.9 & 26.2 & \textbf{31.8} \\
\midrule
\multicolumn{9}{l}{\textit{3. Change number of decoder cross-attention layers.}} \\
\midrule
% \textbf{E3-8, C1-10} & 32.1 & 26.8 & 30.8 & 25.3 & 36.8 & 37.4 & 31.5 & 31.9 \\
\textbf{E3-8, C5-10} & 31.3 & 26.0 & 29.6 & 24.6 & 36.7 & 37.4 & 30.9 & \textbf{32.0} \\
\textbf{E3-8, C3-8}  & 31.4 & 27.6 & 31.7 & 25.7 & 35.4 & 36.9 & 31.5 & 31.7 \\
\textbf{E3-8, C1-6}  & 31.6 & 27.3 & 32.1 & 26.4 & 37.5 & 37.9 & \textbf{32.1} & 31.8 \\
% \textbf{E3-8, C5-6}  & 29.1 & 25.3 & 29.9 & 24.9 & 35.6 & 37.1 & 30.3 & 31.7 \\
\midrule
\multicolumn{9}{l}{\textit{4. Ablation of sharing decoders layers.}} \\
\midrule
\textbf{E3-8, DC3-8} & 30.1 & 23.9 & 30.6 & 25.2 & 35.0 & 34.5 & 29.9 & 31.5 \\
\textbf{E3-8, C3-8} & 31.4 & 27.6 & 31.7 & 25.7 & 35.4 & 36.9 & \textbf{31.5} & \textbf{31.7} \\
\bottomrule
\end{tabular}
}
\label{tab:interlingua}
\end{table}

The first group in \cref{tab:interlingua} conducts an ablation study of the interlingua layer sharing among the encoders. Obviously, without the interlingua layers, there is a gap of as large as 18.2 points in BLEU score on zero-shot translation when compared with the setup with interlingua layers. This clearly shows that encoder interlingua layers are essential to the zero-shot transfer of translation.

The second group in \cref{tab:interlingua} shows zero-shot translation performance when adjusting the language-specific and interlingua partition of the encoder layers under the condition of fixing other settings. Because the low layers of encoders carry more syntactic information, we treat the top layers as interlingua. The average score of zero-shot translation decreases from 30.1 to 26.2 along with the fewer interlingua layers. This trend is consistent with the observation of \cite{conneau2020emerging} in zero-shot cross-lingual transfer learning using pretrained language model that is equivalent to a Transformer encoder. However, the zero-shot difference between the first two rows (\textbf{E1-8,DC3-8} and \textbf{E3-8,DC3-8}) is as small as 0.2 (30.1-29.9), meaning that certain language-specific layers could be equally important to interlingua layers. As a result, in the main experiment and the incremental language expansion, we choose to share the top 6 layers among the encoders and leave the bottom 3 not shared to intentionally give the model more language-specific capacity while not losing the transfer learning ability in zero-shot translation.

The third group in \cref{tab:interlingua} shows the impact of cross-attention to the zero-shot translation. We compare 3 cases in which the cross-attentions happen in the top, middle, and bottom of the decoder stack, respectively. 
The average zero-shot score increases from 30.9 to 32.1 when cross-attention layers change from top (\textbf{C5-10}) to bottom (\textbf{C1-6}). Intuitively, the middle layers of the decoders capture more language-agnostic information than the top and the bottom. So cross-attentions should only happen in the middle stack. Otherwise, the universal representation from encoders could be messed up by language-specific information. However, the experimental result does not match our intuition. \cite{tan2019study} also points out that the bottom layers of decoders capture more language-agnostic information. One possible explanation could be that comparing with language understanding which most likely happens in the bottom layers, language generation in the top stack requires much more capacity to model the complex language phenomenons. Furthermore, conducting cross-attentions in the bottom layers is also better than in all layers (\textbf{E3-8,C1-10} in the first group), which proves that universal representation should not be messed up by language-specific information.


The last group is an ablation test about whether or not sharing decoder layers. The result shows that not sharing decoder layers achieves higher BLEU scores on both zero-shot (31.5 vs. 29.0) and supervised (31.7 vs. 31.5) translation. Interestingly, when comparing \textbf{C1-10} in group 1 and \textbf{DC3-8} in group 2, and assuming that the impact of different cross-attentions is not more than 2 points (based on the results in group 3), we find that decoder sharing is very beneficial in the case of no encoder sharing. Hence, we suspect that the decoder takes the responsibility to tradeoff between keeping the language characteristics and ensuring the representation universality. As a result, we don't share decoder layers in our experiments. However, for applications that depend on finetuning to transfer the decoder to other language generation tasks, it's worth well to further explore the sharing mechanism of the decoder interlingua layers.

\subsubsection{Conclusion}
Summarised by \cref{tab:interlingua}, the recommended configuration for the proposed interlingua structure is sharing the most top layers of the encoders, not sharing any parameter of the decoders, and conducting cross-attentions in the bottom layers of the decoders. Based on this principle, in our main experiment, we use the configuration \textbf{E4-9,C1-6} for the model with 9 encoder layers and 12 decoder layers.


\subsection{Denoising Auto-encoder (DAE) Task Anaysis}

% Because there can be exponentially many combinations considering all the different feasible sets of shared parameters between the multiple decoders, we only select a subset of these combinations based on our preliminary results.

%In the approaches section, we propose to use DAE task as an auxiliary task to help the main translation task (\textbf{MT}). 
In this section, %we empirically prove the effectiveness of the proposed method. 
We conduct an ablation test on the DAE task and compare it with reconstruction task used by previous works \cite{lu2018neural,vazquez2019multilingual}, which we denote as \textbf{AE} (Auto-Encoder) task. 
% Then we give the best choice of noising function illustrated in \cref{fig:noise_function}. 
We use the same experiment settings as in the main experiment unless stated otherwise. The results are shown in \cref{table:dae-analysis}.

\begin{table}[tb]
\caption{Compare the different auxiliary tasks based on zero-shot translation performance. The metric is BLEU. The best scores are in bold.} %Missing scores are marked as ``--''.
\centering
\footnotesize
\resizebox{1\linewidth}{!}{
% \begin{tabular*}{0.99\textwidth}{@{\extracolsep{\fill}} lcc cc cc cc}
\begin{tabular}[b]{l cc cc cc cc}
\toprule
Europarl & \multicolumn{8}{c}{De, Es, Fr $\leftrightarrow$ En} \\
\midrule
\multirow{2}{*}{Direction} & \multicolumn{2}{c}{Fr-De} & \multicolumn{2}{c}{Es-De} & \multicolumn{2}{c}{Es-Fr} & Zero & Parallel \\
& $\leftarrow$ & $\rightarrow$ & $\leftarrow$ & $\rightarrow$ & $\leftarrow$ & $\rightarrow$ & Avg & Avg \\
% \midrule
% \multicolumn{9}{l}{\textit{1. Train model with different auxiliary task.}} \\
\midrule
\textbf{MT}              & 0.44 & 0.35 & 0.94 & 0.41 & 0.82 & 0.41 & 0.56 & 33.26 \\
+\textbf{AE}  & 7.31 & 6.25 & 8.32 & 6.39 & 22.67 & 21.79 & 12.12 & 32.73 \\
% mt+dae(0.1-0.1-0-100) & 27.70 & 23.80 & 30.00 & 24.70 & 37.30 & 37.70 & 30.20 & 33.00 \\
+\textbf{DAE} & 29.30 & 24.60 & 30.13 & 24.48 & 37.39 & 37.59 & 30.58 & 33.29 \\
+\textbf{DAE}+\textbf{Align} & 29.44 & 24.94 & 30.47 & 25.01 & 38.00 & 37.43 & \textbf{30.88} & \textbf{33.37} \\
\bottomrule
\end{tabular}
}
\label{table:dae-analysis}
\end{table}

% In the first group of \cref{table:dae-analysis}, we train model with different auxiliary task. All  settings get similar BLEU points on supervised translation. But the divergence of performance on zero-shot translation are fairly large. 
When trained only using \textbf{MT} task, the zero-shot BLEU score was less than 1.0 for all zero-shot directions. Similar results are also observed by previous works \cite{lu2018neural,vazquez2019multilingual} using language-specific encoders-decoders. 
% Since the training data is En$\leftrightarrow$XX, to language-specific encoders-decoders, the model essentially learns to separate tasks using encoder and decoder without any overlap. For example, training En$\leftrightarrow$Fr uses encoder and decoder respectively. the output of the Fr encoder and the En encoder would never be used by the same decoder, there is no reason for Fr and En source sentences to share the common embedding space, even if they are translations of each other. 
To encourage the model to share the encoder representations across English and non-English source sentences, they added an extra identity language pair (\textbf{AE} task) to joint training. The identity pair forces the non-English source embeddings to be close with English source embeddings. We also experiment with \textbf{AE} task. The result is much better than training only with \textbf{MT} task. The average BLEU score on zero-shot translation is 12.12. 
% The translation between language pairs closed with English, such as Es$\leftrightarrow$Fr, get more boost (Fr$\rightarrow$Es 22.67 and Es$\rightarrow$Fr 21.79) than language pairs distant with English. 
When training with \textbf{DAE} task, we get 30.58 BLEU points on zero-shot translation, which is close to supervised translation (33.29). We conjecture that \textbf{AE} task is too simple for model to learn aligning the non-English sentence representation with English sentence representation. 
% With DAE task, for instance Fr$\rightarrow$Fr, the model can ``translate'' from modified Fr to Fr and translate from En to Fr simultaneously, which makes En and Fr source embedding to be better aligned in a common representation space.
To better understand DAE's ability to align the sentence embedding, we also add an explicit alignment loss for the encoder representations used by previous works \cite{arivazhagan2019missing,zhu2020language,escolano2019bilingual} to improve the zero-shot translation. The result (\textbf{DAE+Align}) shows that, comparing \textbf{DAE} task, explicitly aligning the latent representation only increase 0.3 BLEU points on zero-shot translation. It proves that DAE can provide good enough universal representation for zero-shot translation that makes the representation alignment unnecessary.


\section{Conclusion}
In this paper, to improve the zero-shot translation on language-specific encoders-decoders, we first introduce an interlingua to language-specific encoders-decoders via parameter sharing methods on Transformer layers and selective cross-attention between interlingua and decoder. Then we use denoising auto-encoder task to better align the semantic representation of different languages.
Experiments on the Europarl and MultiUN corpora show that our proposed methods significantly improve the zero-shot translation on language-specific encoders-decoders and achieve competitive or better results than universal encoder-decoder counterparts and pivot-based methods while keeping the superiority of adding a new language without retraining the existing modules.
% , which we proved by an experiment of incremental training. We also empirically examine the optimum configuration choice of our proposed methods and give a detailed analysis. 
% In the future work, we will verified the effectiveness of our approaches on multi-modality and  cross-lingual transfer learning.
% In this paper, we propose an interlingua structure and a multi-task learning scheme to improve the zero-shot translation performance of language-specific encoders-decoders without losing the flexibility of incremental language expansion. Experimental results on Europarl and MultiUN datasets show the effectiveness of the proposed method.


\bibliographystyle{IEEEtran}
\bibliography{IEEEabrv,ijcnn}


\end{document}

IEEEabrv.bib

IEEEabrv.bib
V1.12 (2007/01/11)
Copyright (c) 2002-2007 by Michael Shell
See: http://www.michaelshell.org/
for current contact information.

BibTeX bibliography string definitions of the ABBREVIATED titles of
IEEE journals and magazines and online publications.

This file is designed for bibliography styles that require 
abbreviated titles and is not for use in bibliographies that
require full-length titles.

Support sites:
http://www.michaelshell.org/tex/ieeetran/
http://www.ctan.org/tex-archive/macros/latex/contrib/IEEEtran/
and/or
http://www.ieee.org/

Special thanks to Laura Hyslop and ken Rawson of IEEE for their help
in obtaining the information needed to compile this file. Also,
Volker Kuhlmann and Moritz Borgmann kindly provided some corrections
and additions.

*************************************************************************
Legal Notice:
This code is offered as-is without any warranty either expressed or
implied; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE! 
User assumes all risk.
In no event shall IEEE or any contributor to this code be liable for
any damages or losses, including, but not limited to, incidental,
consequential, or any other damages, resulting from the use or misuse
of any information contained here.

All comments are the opinions of their respective authors and are not
necessarily endorsed by the IEEE.

This work is distributed under the LaTeX Project Public License (LPPL)
( http://www.latex-project.org/ ) version 1.3, and may be freely used,
distributed and modified. A copy of the LPPL, version 1.3, is included
in the base LaTeX documentation of all distributions of LaTeX released
2003/12/01 or later.
Retain all contribution notices and credits.
** Modified files should be clearly indicated as such, including  **
** renaming them and changing author support contact information. **

File list of work: IEEEabrv.bib, IEEEfull.bib, IEEEexample.bib,
                   IEEEtran.bst, IEEEtranS.bst, IEEEtranSA.bst,
                   IEEEtranN.bst, IEEEtranSN.bst, IEEEtran_bst_HOWTO.pdf
*************************************************************************


USAGE:

\bibliographystyle{mybstfile}
\bibliography{IEEEabrv,mybibfile}

where the IEEE titles in the .bib database entries use the strings
defined here. e.g.,


   journal = IEEE_J_AC,


to yield "{IEEE} Trans. Automat. Contr."


IEEE uses abbreviated journal titles in their bibliographies -
this file is suitable for work that is to be submitted to the IEEE.


For work that requires full-length titles, you should use the full
titles provided in the companion file, IEEEfull.bib.


** NOTES **

 1. Journals have been grouped according to subject in order to make it
    easier to locate and extract the definitions for related journals - 
    as most works use references that are confined to a single topic.
    Magazines are listed in straight alphabetical order.

 2. String names are closely based on IEEE's own internal acronyms.
 
 3. Abbreviations follow IEEE's style.

 4. Older, out-of-print IEEE titles are included (but not including titles
    dating prior to IEEE's formation from the IRE and AIEE in 1963).

 5. The following NEW/current journal definitions have been disabled because
    their abbreviations have not yet been verified:
    
    STRING{IEEE_J_CBB        = "{IEEE/ACM} Trans. Comput. Biology Bioinformatics"}
    STRING{IEEE_J_CJECE      = "Canadian J. Elect. Comput. Eng."}
    STRING{IEEE_J_DSC        = "{IEEE} Trans. Dependable Secure Comput."}
    STRING{IEEE_O_DSO        = "{IEEE} Distrib. Syst. Online"}
    
 6. The following OLD journal definitions have been disabled because
    their abbreviations have not yet been found/verified:

    STRING{IEEE_J_BCTV       = "{IEEE} Trans. Broadcast Television Receivers"}
    STRING{IEEE_J_EWS        = "{IEEE} Trans. Eng. Writing Speech"}

If you know what the proper abbreviation is for a string in #5 or #6 above,
email me and I will correct them in the next release.





IEEE Journals 



aerospace and military
@STRING{IEEE_J_AES        = "{IEEE} Trans. Aerosp. Electron. Syst."}
@STRING{IEEE_J_ANE        = "{IEEE} Trans. Aerosp. Navig. Electron."}
@STRING{IEEE_J_ANNE       = "{IEEE} Trans. Aeronaut. Navig. Electron."}
@STRING{IEEE_J_AS         = "{IEEE} Trans. Aerosp."}
@STRING{IEEE_J_AIRE       = "{IEEE} Trans. Airborne Electron."}
@STRING{IEEE_J_MIL        = "{IEEE} Trans. Mil. Electron."}



autos, transportation and vehicles (non-aerospace)
@STRING{IEEE_J_ITS        = "{IEEE} Trans. Intell. Transp. Syst."}
@STRING{IEEE_J_VT         = "{IEEE} Trans. Veh. Technol."}
@STRING{IEEE_J_VC         = "{IEEE} Trans. Veh. Commun."}



circuits, signals, systems, audio and controls
@STRING{IEEE_J_SPL        = "{IEEE} Signal Process. Lett."}
@STRING{IEEE_J_ASSP       = "{IEEE} Trans. Acoust., Speech, Signal Process."}
@STRING{IEEE_J_AU         = "{IEEE} Trans. Audio"}
@STRING{IEEE_J_AUEA       = "{IEEE} Trans. Audio Electroacoust."}
@STRING{IEEE_J_AC         = "{IEEE} Trans. Autom. Control"}
@STRING{IEEE_J_CAS        = "{IEEE} Trans. Circuits Syst."}
@STRING{IEEE_J_CASVT      = "{IEEE} Trans. Circuits Syst. Video Technol."}
@STRING{IEEE_J_CASI       = "{IEEE} Trans. Circuits Syst. {I}"}
@STRING{IEEE_J_CASII      = "{IEEE} Trans. Circuits Syst. {II}"}
in 2004 CASI and CASII renamed part title to CASI_RP and CASII_EB, respectively.
@STRING{IEEE_J_CASI_RP    = "{IEEE} Trans. Circuits Syst. {I}"}
@STRING{IEEE_J_CASII_EB   = "{IEEE} Trans. Circuits Syst. {II}"}
@STRING{IEEE_J_CT         = "{IEEE} Trans. Circuit Theory"}
@STRING{IEEE_J_CST        = "{IEEE} Trans. Control Syst. Technol."}
@STRING{IEEE_J_SP         = "{IEEE} Trans. Signal Process."}
@STRING{IEEE_J_SU         = "{IEEE} Trans. Sonics Ultrason."}
@STRING{IEEE_J_SAP        = "{IEEE} Trans. Speech Audio Process."}
@STRING{IEEE_J_UE         = "{IEEE} Trans. Ultrason. Eng."}
@STRING{IEEE_J_UFFC       = "{IEEE} Trans. Ultrason., Ferroelectr., Freq. Control"}



communications
@STRING{IEEE_J_COML       = "{IEEE} Commun. Lett."}
@STRING{IEEE_J_JSAC       = "{IEEE} J. Sel. Areas Commun."}
@STRING{IEEE_J_COM        = "{IEEE} Trans. Commun."}
@STRING{IEEE_J_COMT       = "{IEEE} Trans. Commun. Technol."}
@STRING{IEEE_J_WCOM       = "{IEEE} Trans. Wireless Commun."}



components, packaging and manufacturing
@STRING{IEEE_J_ADVP       = "{IEEE} Trans. Adv. Packag."}
@STRING{IEEE_J_CHMT       = "{IEEE} Trans. Compon., Hybrids, Manuf. Technol."}
@STRING{IEEE_J_CPMTA      = "{IEEE} Trans. Compon., Packag., Manuf. Technol. {A}"}
@STRING{IEEE_J_CPMTB      = "{IEEE} Trans. Compon., Packag., Manuf. Technol. {B}"}
@STRING{IEEE_J_CPMTC      = "{IEEE} Trans. Compon., Packag., Manuf. Technol. {C}"}
@STRING{IEEE_J_CAPT       = "{IEEE} Trans. Compon. Packag. Technol."}
@STRING{IEEE_J_CAPTS      = "{IEEE} Trans. Compon. Packag. Technol."}
@STRING{IEEE_J_CPART      = "{IEEE} Trans. Compon. Parts"}
@STRING{IEEE_J_EPM        = "{IEEE} Trans. Electron. Packag. Manuf."}
@STRING{IEEE_J_MFT        = "{IEEE} Trans. Manuf. Technol."}
@STRING{IEEE_J_PHP        = "{IEEE} Trans. Parts, Hybrids, Packag."}
@STRING{IEEE_J_PMP        = "{IEEE} Trans. Parts, Mater., Packag."}



CAD
@STRING{IEEE_J_TCAD       = "{IEEE} J. Technol. Comput. Aided Design"}
@STRING{IEEE_J_CAD        = "{IEEE} Trans. Comput.-Aided Design Integr. Circuits Syst."}



coding, data, information, knowledge
@STRING{IEEE_J_IT         = "{IEEE} Trans. Inf. Theory"}
@STRING{IEEE_J_KDE        = "{IEEE} Trans. Knowl. Data Eng."}



computers, computation, networking and software
@STRING{IEEE_J_C          = "{IEEE} Trans. Comput."}
@STRING{IEEE_J_CAL        = "{IEEE} Comput. Archit. Lett."}
disabled till definition is verified
STRING{IEEE_J_DSC         = "{IEEE} Trans. Dependable Secure Comput."}
@STRING{IEEE_J_ECOMP      = "{IEEE} Trans. Electron. Comput."}
@STRING{IEEE_J_EVC        = "{IEEE} Trans. Evol. Comput."}
@STRING{IEEE_J_FUZZ       = "{IEEE} Trans. Fuzzy Syst."}
@STRING{IEEE_J_IFS        = "{IEEE} Trans. Inf. Forensics Security"}
@STRING{IEEE_J_MC         = "{IEEE} Trans. Mobile Comput."}
@STRING{IEEE_J_NET        = "{IEEE/ACM} Trans. Netw."}
@STRING{IEEE_J_NN         = "{IEEE} Trans. Neural Netw."}
@STRING{IEEE_J_PDS        = "{IEEE} Trans. Parallel Distrib. Syst."}
@STRING{IEEE_J_SE         = "{IEEE} Trans. Softw. Eng."}



computer graphics, imaging, and multimedia
@STRING{IEEE_J_JDT        = "{IEEE/OSA} J. Display Technol."}
@STRING{IEEE_J_IP         = "{IEEE} Trans. Image Process."}
@STRING{IEEE_J_MM         = "{IEEE} Trans. Multimedia"}
@STRING{IEEE_J_VCG        = "{IEEE} Trans. Vis. Comput. Graphics"}



cybernetics, ergonomics, robots, man-machine, and automation
@STRING{IEEE_J_ASE        = "{IEEE} Trans. Autom. Sci. Eng."}
@STRING{IEEE_J_JRA        = "{IEEE} J. Robot. Autom."}
@STRING{IEEE_J_HFE        = "{IEEE} Trans. Hum. Factors Electron."}
@STRING{IEEE_J_MMS        = "{IEEE} Trans. Man-Mach. Syst."}
@STRING{IEEE_J_PAMI       = "{IEEE} Trans. Pattern Anal. Mach. Intell."}
in 1989 JRA became RA
in August 2004, RA split into ASE and RO
@STRING{IEEE_J_RA         = "{IEEE} Trans. Robot. Autom."}
@STRING{IEEE_J_RO         = "{IEEE} Trans. Robot."}
@STRING{IEEE_J_SMC        = "{IEEE} Trans. Syst., Man, Cybern."}
@STRING{IEEE_J_SMCA       = "{IEEE} Trans. Syst., Man, Cybern. {A}"}
@STRING{IEEE_J_SMCB       = "{IEEE} Trans. Syst., Man, Cybern. {B}"}
@STRING{IEEE_J_SMCC       = "{IEEE} Trans. Syst., Man, Cybern. {C}"}
@STRING{IEEE_J_SSC        = "{IEEE} Trans. Syst. Sci. Cybern."}



earth, wind, fire and water
@STRING{IEEE_J_GE         = "{IEEE} Trans. Geosci. Electron."}
@STRING{IEEE_J_GRS        = "{IEEE} Trans. Geosci. Remote Sens."}
@STRING{IEEE_J_GRSL       = "{IEEE} Geosci. Remote Sens. Lett."}
@STRING{IEEE_J_OE         = "{IEEE} J. Ocean. Eng."}



education, engineering, history, IEEE, professional
disabled till definition is verified
STRING{IEEE_J_CJECE       = "Canadian J. Elect. Comput. Eng."}
@STRING{IEEE_J_PROC       = "Proc. {IEEE}"}
@STRING{IEEE_J_EDU        = "{IEEE} Trans. Educ."}
@STRING{IEEE_J_EM         = "{IEEE} Trans. Eng. Manag."}
disabled till definition is verified
STRING{IEEE_J_EWS         = "{IEEE} Trans. Eng. Writing Speech"}
@STRING{IEEE_J_PC         = "{IEEE} Trans. Prof. Commun."}



electromagnetics, antennas, EMI, magnetics and microwave
@STRING{IEEE_J_AWPL       = "{IEEE} Antennas Wireless Propag. Lett."}
@STRING{IEEE_J_MGWL       = "{IEEE} Microw. Guided Wave Lett."}
IEEE seems to want "Compon." here, not "Comp."
@STRING{IEEE_J_MWCL       = "{IEEE} Microw. Wireless Compon. Lett."}
@STRING{IEEE_J_AP         = "{IEEE} Trans. Antennas Propag."}
@STRING{IEEE_J_EMC        = "{IEEE} Trans. Electromagn. Compat."}
@STRING{IEEE_J_MAG        = "{IEEE} Trans. Magn."}
@STRING{IEEE_J_MTT        = "{IEEE} Trans. Microw. Theory Tech."}
@STRING{IEEE_J_RFI        = "{IEEE} Trans. Radio Freq. Interference"}
@STRING{IEEE_J_TJMJ       = "{IEEE} Transl. J. Magn. Jpn."}



energy and power
@STRING{IEEE_J_EC         = "{IEEE} Trans. Energy Convers."}
@STRING{IEEE_J_PEL        = "{IEEE} Power Electron. Lett."}
@STRING{IEEE_J_PWRAS      = "{IEEE} Trans. Power App. Syst."}
@STRING{IEEE_J_PWRD       = "{IEEE} Trans. Power Del."}
@STRING{IEEE_J_PWRE       = "{IEEE} Trans. Power Electron."}
@STRING{IEEE_J_PWRS       = "{IEEE} Trans. Power Syst."}



industrial, commercial and consumer
@STRING{IEEE_J_APPIND     = "{IEEE} Trans. Appl. Ind."}
@STRING{IEEE_J_BC         = "{IEEE} Trans. Broadcast."}
disabled till definition is verified
STRING{IEEE_J_BCTV        = "{IEEE} Trans. Broadcast Television Receivers"}
@STRING{IEEE_J_CE         = "{IEEE} Trans. Consum. Electron."}
@STRING{IEEE_J_IE         = "{IEEE} Trans. Ind. Electron."}
@STRING{IEEE_J_IECI       = "{IEEE} Trans. Ind. Electron. Contr. Instrum."}
@STRING{IEEE_J_IA         = "{IEEE} Trans. Ind. Appl."}
@STRING{IEEE_J_IGA        = "{IEEE} Trans. Ind. Gen. Appl."}
@STRING{IEEE_J_IINF       = "{IEEE} Trans. Ind. Informat."}
@STRING{IEEE_J_PSE        = "{IEEE} J. Product Safety Eng."}



instrumentation and measurement
@STRING{IEEE_J_IM         = "{IEEE} Trans. Instrum. Meas."}



insulation and materials
@STRING{IEEE_J_JEM        = "{IEEE/TMS} J. Electron. Mater."}
@STRING{IEEE_J_DEI        = "{IEEE} Trans. Dielectr. Electr. Insul."}
@STRING{IEEE_J_EI         = "{IEEE} Trans. Electr. Insul."}



mechanical
@STRING{IEEE_J_MECH       = "{IEEE/ASME} Trans. Mechatronics"}
@STRING{IEEE_J_MEMS       = "J. Microelectromech. Syst."}



medical and biological
@STRING{IEEE_J_BME        = "{IEEE} Trans. Biomed. Eng."}
Note: The B-ME journal later dropped the hyphen and became the BME.
@STRING{IEEE_J_B-ME       = "{IEEE} Trans. Bio-Med. Eng."}
@STRING{IEEE_J_BMELC      = "{IEEE} Trans. Bio-Med. Electron."}
disabled till definition is verified
STRING{IEEE_J_CBB         = "{IEEE/ACM} Trans. Comput. Biology Bioinformatics"}
@STRING{IEEE_J_ITBM       = "{IEEE} Trans. Inf. Technol. Biomed."}
@STRING{IEEE_J_ME         = "{IEEE} Trans. Med. Electron."}
@STRING{IEEE_J_MI         = "{IEEE} Trans. Med. Imag."}
@STRING{IEEE_J_NB         = "{IEEE} Trans. Nanobiosci."}
@STRING{IEEE_J_NSRE       = "{IEEE} Trans. Neural Syst. Rehabil. Eng."}
@STRING{IEEE_J_RE         = "{IEEE} Trans. Rehabil. Eng."}



 optics, lightwave and photonics
@STRING{IEEE_J_PTL        = "{IEEE} Photon. Technol. Lett."}
@STRING{IEEE_J_JLT        = "J. Lightw. Technol."}



physics, electrons, nanotechnology, nuclear and quantum electronics
@STRING{IEEE_J_EDL        = "{IEEE} Electron Device Lett."}
@STRING{IEEE_J_JQE        = "{IEEE} J. Quantum Electron."}
@STRING{IEEE_J_JSTQE      = "{IEEE} J. Sel. Topics Quantum Electron."}
@STRING{IEEE_J_ED         = "{IEEE} Trans. Electron Devices"}
@STRING{IEEE_J_NANO       = "{IEEE} Trans. Nanotechnol."}
@STRING{IEEE_J_NS         = "{IEEE} Trans. Nucl. Sci."}
@STRING{IEEE_J_PS         = "{IEEE} Trans. Plasma Sci."}



reliability
IEEE seems to want "Mat." here, not "Mater."
@STRING{IEEE_J_DMR        = "{IEEE} Trans. Device Mater. Rel."}
@STRING{IEEE_J_R          = "{IEEE} Trans. Rel."}



semiconductors, superconductors, electrochemical and solid state
@STRING{IEEE_J_ESSL       = "{IEEE/ECS} Electrochem. Solid-State Lett."}
@STRING{IEEE_J_JSSC       = "{IEEE} J. Solid-State Circuits"}
@STRING{IEEE_J_ASC        = "{IEEE} Trans. Appl. Supercond."}
@STRING{IEEE_J_SM         = "{IEEE} Trans. Semicond. Manuf."}



sensors
@STRING{IEEE_J_SENSOR     = "{IEEE} Sensors J."}



VLSI
@STRING{IEEE_J_VLSI       = "{IEEE} Trans. {VLSI} Syst."}






IEEE Magazines 



@STRING{IEEE_M_AES        = "{IEEE} Aerosp. Electron. Syst. Mag."}
@STRING{IEEE_M_HIST       = "{IEEE} Ann. Hist. Comput."}
@STRING{IEEE_M_AP         = "{IEEE} Antennas Propag. Mag."}
@STRING{IEEE_M_ASSP       = "{IEEE} {ASSP} Mag."}
@STRING{IEEE_M_CD         = "{IEEE} Circuits Devices Mag."}
@STRING{IEEE_M_CAS        = "{IEEE} Circuits Syst. Mag."}
@STRING{IEEE_M_COM        = "{IEEE} Commun. Mag."}
@STRING{IEEE_M_COMSOC     = "{IEEE} Commun. Soc. Mag."}
@STRING{IEEE_M_CIM        = "{IEEE} Comput. Intell. Mag."}
CSEM changed to CSE in 1999
@STRING{IEEE_M_CSE        = "{IEEE} Comput. Sci. Eng."}
@STRING{IEEE_M_CSEM       = "{IEEE} Comput. Sci. Eng. Mag."}
@STRING{IEEE_M_C          = "{IEEE} Computer"}
@STRING{IEEE_M_CAP        = "{IEEE} Comput. Appl. Power"}
@STRING{IEEE_M_CGA        = "{IEEE} Comput. Graph. Appl."}
@STRING{IEEE_M_CONC       = "{IEEE} Concurrency"}
@STRING{IEEE_M_CS         = "{IEEE} Control Syst. Mag."}
@STRING{IEEE_M_DTC        = "{IEEE} Des. Test. Comput."}
@STRING{IEEE_M_EI         = "{IEEE} Electr. Insul. Mag."}
@STRING{IEEE_M_ETR        = "{IEEE} ElectroTechnol. Rev."}
@STRING{IEEE_M_EMB        = "{IEEE} Eng. Med. Biol. Mag."}
@STRING{IEEE_M_EMR        = "{IEEE} Eng. Manag. Rev."}
@STRING{IEEE_M_EXP        = "{IEEE} Expert"}
@STRING{IEEE_M_IA         = "{IEEE} Ind. Appl. Mag."}
@STRING{IEEE_M_IM         = "{IEEE} Instrum. Meas. Mag."}
@STRING{IEEE_M_IS         = "{IEEE} Intell. Syst."}
@STRING{IEEE_M_IC         = "{IEEE} Internet Comput."}
@STRING{IEEE_M_ITP        = "{IEEE} {IT} Prof."}
@STRING{IEEE_M_MICRO      = "{IEEE} Micro"}
@STRING{IEEE_M_MW         = "{IEEE} Microw. Mag."}
@STRING{IEEE_M_MM         = "{IEEE} Multimedia"}
@STRING{IEEE_M_NET        = "{IEEE} Netw."}
IEEE's editorial manual lists "Pers. Commun.", 
but "Personal Commun. Mag." seems to be what is used in the journals
@STRING{IEEE_M_PCOM       = "{IEEE} Personal Commun. Mag."}
@STRING{IEEE_M_POT        = "{IEEE} Potentials"}
CAP and PER merged to form PE in 2003
@STRING{IEEE_M_PE         = "{IEEE} Power Energy Mag."}
@STRING{IEEE_M_PER        = "{IEEE} Power Eng. Rev."}
@STRING{IEEE_M_PVC        = "{IEEE} Pervasive Comput."}
@STRING{IEEE_M_RA         = "{IEEE} Robot. Autom. Mag."}
@STRING{IEEE_M_SAP        = "{IEEE} Security Privacy"}
@STRING{IEEE_M_SP         = "{IEEE} Signal Process. Mag."}
@STRING{IEEE_M_S          = "{IEEE} Softw."}
@STRING{IEEE_M_SPECT      = "{IEEE} Spectr."}
@STRING{IEEE_M_TS         = "{IEEE} Technol. Soc. Mag."}
@STRING{IEEE_M_VT         = "{IEEE} Veh. Technol. Mag."}
@STRING{IEEE_M_WC         = "{IEEE} Wireless Commun. Mag."}
@STRING{IEEE_M_TODAY      = "Today's Engineer"}






IEEE Online Publications 



@STRING{IEEE_O_CSTO        = "{IEEE} Commun. Surveys Tuts."}
disabled till definition is verified
STRING{IEEE_O_DSO          = "{IEEE} Distrib. Syst. Online"}





--
EOF

conference_101719.bbl
% Generated by IEEEtran.bst, version: 1.12 (2007/01/11)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
  \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl

\bibitem{johnson2017google}
M.~Johnson, M.~Schuster, Q.~V. Le, M.~Krikun, Y.~Wu, Z.~Chen, N.~Thorat,
  F.~Vi{\'e}gas, M.~Wattenberg, G.~Corrado \emph{et~al.}, ``Google’s
  multilingual neural machine translation system: Enabling zero-shot
  translation,'' \emph{Transactions of the Association for Computational
  Linguistics}, vol.~5, pp. 339--351, 2017.

\bibitem{gu2019improved}
J.~Gu, Y.~Wang, K.~Cho, and V.~O. Li, ``Improved zero-shot neural machine
  translation via ignoring spurious correlations,'' in \emph{Proceedings of the
  57th Annual Meeting of the Association for Computational Linguistics}, 2019,
  pp. 1258--1268.

\bibitem{ji2020cross}
B.~Ji, Z.~Zhang, X.~Duan, M.~Zhang, B.~Chen, and W.~Luo, ``Cross-lingual
  pre-training based transfer for zero-shot neural machine translation,'' in
  \emph{Proceedings of the AAAI Conference on Artificial Intelligence},
  vol.~34, 2020, pp. 115--122.

\bibitem{al2019consistency}
M.~Al-Shedivat and A.~Parikh, ``Consistency by agreement in zero-shot neural
  machine translation,'' in \emph{Proceedings of the 2019 Conference of the
  North American Chapter of the Association for Computational Linguistics:
  Human Language Technologies, Volume 1 (Long and Short Papers)}, 2019, pp.
  1184--1197.

\bibitem{arivazhagan2019missing}
N.~Arivazhagan, A.~Bapna, O.~Firat, R.~Aharoni, M.~Johnson, and W.~Macherey,
  ``The missing ingredient in zero-shot neural machine translation,''
  \emph{arXiv preprint arXiv:1903.07091}, 2019.

\bibitem{sestorain2018zero}
L.~Sestorain, M.~Ciaramita, C.~Buck, and T.~Hofmann, ``Zero-shot dual machine
  translation,'' \emph{arXiv preprint arXiv:1805.10338}, 2018.

\bibitem{he2016dual}
D.~He, Y.~Xia, T.~Qin, L.~Wang, N.~Yu, T.-Y. Liu, and W.-Y. Ma, ``Dual learning
  for machine translation,'' in \emph{Advances in neural information processing
  systems}, 2016, pp. 820--828.

\bibitem{firat2016multi}
O.~Firat, K.~Cho, and Y.~Bengio, ``Multi-way, multilingual neural machine
  translation with a shared attention mechanism,'' in \emph{Proceedings of the
  2016 Conference of the North American Chapter of the Association for
  Computational Linguistics: Human Language Technologies}, 2016, pp. 866--875.

\bibitem{lu2018neural}
Y.~Lu, P.~Keung, F.~Ladhak, V.~Bhardwaj, S.~Zhang, and J.~Sun, ``A neural
  interlingua for multilingual machine translation,'' in \emph{Proceedings of
  the Third Conference on Machine Translation: Research Papers}, 2018, pp.
  84--92.

\bibitem{vazquez2019multilingual}
R.~V{\'a}zquez, A.~Raganato, J.~Tiedemann, and M.~Creutz, ``Multilingual nmt
  with a language-independent attention bridge,'' in \emph{Proceedings of the
  4th Workshop on Representation Learning for NLP (RepL4NLP-2019)}, 2019, pp.
  33--39.

\bibitem{escolano2019bilingual}
C.~Escolano, M.~R. Costa-juss{\`a}, and J.~A. Fonollosa, ``From bilingual to
  multilingual neural machine translation by incremental training,'' in
  \emph{Proceedings of the 57th Annual Meeting of the Association for
  Computational Linguistics: Student Research Workshop}, 2019, pp. 236--242.

\bibitem{escolano2020multilingual}
C.~Escolano, M.~R. Costa-juss{\`a}, J.~A. Fonollosa, and M.~Artetxe,
  ``Multilingual machine translation: Closing the gap between shared and
  language-specific encoder-decoders,'' \emph{arXiv preprint arXiv:2004.06575},
  2020.

\bibitem{escolano2020training}
------, ``Training multilingual machine translation by alternately freezing
  language-specific encoders-decoders,'' \emph{arXiv preprint
  arXiv:2006.01594}, 2020.

\bibitem{vaswani2017attention}
A.~Vaswani, N.~Shazeer, N.~Parmar, J.~Uszkoreit, L.~Jones, A.~N. Gomez,
  {\L}.~Kaiser, and I.~Polosukhin, ``Attention is all you need,'' in
  \emph{Advances in neural information processing systems}, 2017, pp.
  5998--6008.

\bibitem{zhu2020language}
C.~Zhu, H.~Yu, S.~Cheng, and W.~Luo, ``Language-aware interlingua for
  multilingual neural machine translation,'' in \emph{Proceedings of the 58th
  Annual Meeting of the Association for Computational Linguistics}, 2020, pp.
  1650--1655.

\bibitem{blackwood2018multilingual}
G.~Blackwood, M.~Ballesteros, and T.~Ward, ``Multilingual neural machine
  translation with task-specific attention,'' in \emph{Proceedings of the 27th
  International Conference on Computational Linguistics}, 2018, pp. 3112--3122.

\bibitem{zaremoodi2018adaptive}
P.~Zaremoodi, W.~Buntine, and G.~Haffari, ``Adaptive knowledge sharing in
  multi-task learning: Improving low-resource neural machine translation,'' in
  \emph{Proceedings of the 56th Annual Meeting of the Association for
  Computational Linguistics (Volume 2: Short Papers)}, 2018, pp. 656--661.

\bibitem{platanios2018contextual}
E.~A. Platanios, M.~Sachan, G.~Neubig, and T.~Mitchell, ``Contextual parameter
  generation for universal neural machine translation,'' in \emph{Proceedings
  of the 2018 Conference on Empirical Methods in Natural Language Processing},
  2018, pp. 425--435.

\bibitem{sachan2018parameter}
D.~Sachan and G.~Neubig, ``Parameter sharing methods for multilingual
  self-attentional translation models,'' in \emph{Proceedings of the Third
  Conference on Machine Translation: Research Papers}, 2018, pp. 261--271.

\bibitem{conneau2020emerging}
A.~Conneau, S.~Wu, H.~Li, L.~Zettlemoyer, and V.~Stoyanov, ``Emerging
  cross-lingual structure in pretrained language models,'' in \emph{Proceedings
  of the 58th Annual Meeting of the Association for Computational Linguistics},
  2020, pp. 6022--6034.

\bibitem{artetxe2020cross}
M.~Artetxe, S.~Ruder, and D.~Yogatama, ``On the cross-lingual transferability
  of monolingual representations,'' in \emph{Proceedings of the 58th Annual
  Meeting of the Association for Computational Linguistics}, 2020, pp.
  4623--4637.

\bibitem{tan2019study}
X.~Tan, Y.~Leng, J.~Chen, Y.~Ren, T.~Qin, and T.-Y. Liu, ``A study of
  multilingual neural machine translation,'' \emph{arXiv preprint
  arXiv:1912.11625}, 2019.

\bibitem{koehn2005europarl}
P.~Koehn, ``Europarl: A parallel corpus for statistical machine translation,''
  in \emph{MT summit}, vol.~5.\hskip 1em plus 0.5em minus 0.4em\relax Citeseer,
  2005, pp. 79--86.

\bibitem{eisele2010multiun}
A.~Eisele and Y.~Chen, ``Multiun: A multilingual corpus from united nation
  documents.'' in \emph{LREC}, 2010.

\bibitem{papineni2002bleu}
K.~Papineni, S.~Roukos, T.~Ward, and W.-J. Zhu, ``Bleu: a method for automatic
  evaluation of machine translation,'' in \emph{Proceedings of the 40th annual
  meeting of the Association for Computational Linguistics}, 2002, pp.
  311--318.

\bibitem{post2018call}
M.~Post, ``A call for clarity in reporting bleu scores,'' in \emph{Proceedings
  of the Third Conference on Machine Translation: Research Papers}, 2018, pp.
  186--191.

\bibitem{kudo2018sentencepiece}
T.~Kudo and J.~Richardson, ``Sentencepiece: A simple and language independent
  subword tokenizer and detokenizer for neural text processing,'' in
  \emph{Proceedings of the 2018 Conference on Empirical Methods in Natural
  Language Processing: System Demonstrations}, 2018, pp. 66--71.

\bibitem{kudo2018subword}
T.~Kudo, ``Subword regularization: Improving neural network translation models
  with multiple subword candidates,'' in \emph{Proceedings of the 56th Annual
  Meeting of the Association for Computational Linguistics (Volume 1: Long
  Papers)}, 2018, pp. 66--75.

\bibitem{press2017using}
O.~Press and L.~Wolf, ``Using the output embedding to improve language
  models,'' in \emph{Proceedings of the 15th Conference of the European Chapter
  of the Association for Computational Linguistics: Volume 2, Short Papers},
  2017, pp. 157--163.

\bibitem{liu2019variance}
L.~Liu, H.~Jiang, P.~He, W.~Chen, X.~Liu, J.~Gao, and J.~Han, ``On the variance
  of the adaptive learning rate and beyond,'' in \emph{International Conference
  on Learning Representations}, 2019.

\bibitem{hochreiter1997long}
S.~Hochreiter and J.~Schmidhuber, ``Long short-term memory,'' \emph{Neural
  computation}, vol.~9, no.~8, pp. 1735--1780, 1997.

\bibitem{lakew2018comparison}
S.~M. Lakew, M.~Cettolo, and M.~Federico, ``A comparison of transformer and
  recurrent neural networks on multilingual neural machine translation,'' in
  \emph{Proceedings of the 27th International Conference on Computational
  Linguistics}, 2018, pp. 641--652.

\bibitem{sennrich2016improving}
R.~Sennrich, B.~Haddow, and A.~Birch, ``Improving neural machine translation
  models with monolingual data,'' in \emph{Proceedings of the 54th Annual
  Meeting of the Association for Computational Linguistics (Volume 1: Long
  Papers)}, 2016, pp. 86--96.

\end{thebibliography}

ijcnn.bib
@article{johnson2017google,
  title={Google’s multilingual neural machine translation system: Enabling zero-shot translation},
  author={Johnson, Melvin and Schuster, Mike and Le, Quoc V and Krikun, Maxim and Wu, Yonghui and Chen, Zhifeng and Thorat, Nikhil and Vi{\'e}gas, Fernanda and Wattenberg, Martin and Corrado, Greg and others},
  journal={Transactions of the Association for Computational Linguistics},
  volume={5},
  pages={339--351},
  year={2017},
  publisher={MIT Press}
}

@inproceedings{gu2019improved,
  title={Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations},
  author={Gu, Jiatao and Wang, Yong and Cho, Kyunghyun and Li, Victor OK},
  booktitle={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  pages={1258--1268},
  year={2019}
}

@inproceedings{ji2020cross,
  title={Cross-Lingual Pre-Training Based Transfer for Zero-Shot Neural Machine Translation},
  author={Ji, Baijun and Zhang, Zhirui and Duan, Xiangyu and Zhang, Min and Chen, Boxing and Luo, Weihua},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={34},
  pages={115--122},
  year={2020}
}

@inproceedings{al2019consistency,
  title={Consistency by Agreement in Zero-Shot Neural Machine Translation},
  author={Al-Shedivat, Maruan and Parikh, Ankur},
  booktitle={Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  pages={1184--1197},
  year={2019}
}

@article{arivazhagan2019missing,
  title={The missing ingredient in zero-shot neural machine translation},
  author={Arivazhagan, Naveen and Bapna, Ankur and Firat, Orhan and Aharoni, Roee and Johnson, Melvin and Macherey, Wolfgang},
  journal={arXiv preprint arXiv:1903.07091},
  year={2019}
}

@article{sestorain2018zero,
  title={Zero-shot dual machine translation},
  author={Sestorain, Lierni and Ciaramita, Massimiliano and Buck, Christian and Hofmann, Thomas},
  journal={arXiv preprint arXiv:1805.10338},
  year={2018}
}

@inproceedings{he2016dual,
  title={Dual learning for machine translation},
  author={He, Di and Xia, Yingce and Qin, Tao and Wang, Liwei and Yu, Nenghai and Liu, Tie-Yan and Ma, Wei-Ying},
  booktitle={Advances in neural information processing systems},
  pages={820--828},
  year={2016}
}

@inproceedings{sutskever2014sequence,
  title={Sequence to sequence learning with neural networks},
  author={Sutskever, Ilya and Vinyals, Oriol and Le, Quoc V},
  booktitle={Advances in neural information processing systems},
  pages={3104--3112},
  year={2014}
}

@inproceedings{bahdanau2015neural,
  title={Neural machine translation by jointly learning to align and translate},
  author={Bahdanau, Dzmitry and Cho, Kyung Hyun and Bengio, Yoshua},
  booktitle={3rd International Conference on Learning Representations, ICLR 2015},
  year={2015}
}

@inproceedings{vaswani2017attention,
  title={Attention is all you need},
  author={Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, {\L}ukasz and Polosukhin, Illia},
  booktitle={Advances in neural information processing systems},
  pages={5998--6008},
  year={2017}
}

@article{escolano2020training,
  title={Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoders},
  author={Escolano, Carlos and Costa-juss{\`a}, Marta R and Fonollosa, Jos{\'e} AR and Artetxe, Mikel},
  journal={arXiv preprint arXiv:2006.01594},
  year={2020}
}

@article{escolano2020multilingual,
  title={Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoders},
  author={Escolano, Carlos and Costa-juss{\`a}, Marta R and Fonollosa, Jos{\'e} AR and Artetxe, Mikel},
  journal={arXiv preprint arXiv:2004.06575},
  year={2020}
}


@inproceedings{escolano2019bilingual,
  title={From Bilingual to Multilingual Neural Machine Translation by Incremental Training},
  author={Escolano, Carlos and Costa-juss{\`a}, Marta R and Fonollosa, Jos{\'e} AR},
  booktitle={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop},
  pages={236--242},
  year={2019}
}

@inproceedings{firat2016multi,
  title={Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism},
  author={Firat, Orhan and Cho, Kyunghyun and Bengio, Yoshua},
  booktitle={Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  pages={866--875},
  year={2016}
}

@inproceedings{lu2018neural,
  title={A neural interlingua for multilingual machine translation},
  author={Lu, Yichao and Keung, Phillip and Ladhak, Faisal and Bhardwaj, Vikas and Zhang, Shaonan and Sun, Jason},
  booktitle={Proceedings of the Third Conference on Machine Translation: Research Papers},
  pages={84--92},
  year={2018}
}

@inproceedings{vazquez2019multilingual,
  title={Multilingual NMT with a Language-Independent Attention Bridge},
  author={V{\'a}zquez, Ra{\'u}l and Raganato, Alessandro and Tiedemann, J{\"o}rg and Creutz, Mathias},
  booktitle={Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)},
  pages={33--39},
  year={2019}
}

@inproceedings{zhu2020language,
  title={Language-aware Interlingua for Multilingual Neural Machine Translation},
  author={Zhu, Changfeng and Yu, Heng and Cheng, Shanbo and Luo, Weihua},
  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  pages={1650--1655},
  year={2020}
}

@inproceedings{koehn2005europarl,
  title={Europarl: A parallel corpus for statistical machine translation},
  author={Koehn, Philipp},
  booktitle={MT summit},
  volume={5},
  pages={79--86},
  year={2005},
  organization={Citeseer}
}

@inproceedings{eisele2010multiun,
  title={MultiUN: A Multilingual Corpus from United Nation Documents.},
  author={Eisele, Andreas and Chen, Yu},
  booktitle={LREC},
  year={2010}
}

@inproceedings{conneau2020emerging,
  title={Emerging Cross-lingual Structure in Pretrained Language Models},
  author={Conneau, Alexis and Wu, Shijie and Li, Haoran and Zettlemoyer, Luke and Stoyanov, Veselin},
  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  pages={6022--6034},
  year={2020}
}

@inproceedings{artetxe2020cross,
  title={On the Cross-lingual Transferability of Monolingual Representations},
  author={Artetxe, Mikel and Ruder, Sebastian and Yogatama, Dani},
  booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  pages={4623--4637},
  year={2020}
}

@article{tan2019study,
  title={A Study of Multilingual Neural Machine Translation},
  author={Tan, Xu and Leng, Yichong and Chen, Jiale and Ren, Yi and Qin, Tao and Liu, Tie-Yan},
  journal={arXiv preprint arXiv:1912.11625},
  year={2019}
}

@article{liu2020multilingual,
  title={Multilingual denoising pre-training for neural machine translation},
  author={Liu, Yinhan and Gu, Jiatao and Goyal, Naman and Li, Xian and Edunov, Sergey and Ghazvininejad, Marjan and Lewis, Mike and Zettlemoyer, Luke},
  journal={Transactions of the Association for Computational Linguistics},
  volume={8},
  pages={726--742},
  year={2020},
  publisher={MIT Press}
}

@inproceedings{papineni2002bleu,
  title={BLEU: a method for automatic evaluation of machine translation},
  author={Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing},
  booktitle={Proceedings of the 40th annual meeting of the Association for Computational Linguistics},
  pages={311--318},
  year={2002}
}

@inproceedings{post2018call,
  title={A Call for Clarity in Reporting BLEU Scores},
  author={Post, Matt},
  booktitle={Proceedings of the Third Conference on Machine Translation: Research Papers},
  pages={186--191},
  year={2018}
}

@inproceedings{kudo2018sentencepiece,
  title={SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing},
  author={Kudo, Taku and Richardson, John},
  booktitle={Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  pages={66--71},
  year={2018}
}

@inproceedings{sennrich2016neural,
  title={Neural Machine Translation of Rare Words with Subword Units},
  author={Sennrich, Rico and Haddow, Barry and Birch, Alexandra},
  booktitle={Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={1715--1725},
  year={2016}
}

@inproceedings{kudo2018subword,
  title={Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates},
  author={Kudo, Taku},
  booktitle={Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={66--75},
  year={2018}
}

@article{raffel2020exploring,
  title={Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
  author={Raffel, Colin and Shazeer, Noam and Roberts, Adam and Lee, Katherine and Narang, Sharan and Matena, Michael and Zhou, Yanqi and Li, Wei and Liu, Peter J},
  journal={Journal of Machine Learning Research},
  volume={21},
  pages={1--67},
  year={2020}
}

@inproceedings{kingma2015adam,
  title={Adam: A Method for Stochastic Optimization},
  author={Kingma, Diederik P and Ba, Jimmy},
  booktitle={ICLR (Poster)},
  year={2015}
}

@inproceedings{press2017using,
  title={Using the Output Embedding to Improve Language Models},
  author={Press, Ofir and Wolf, Lior},
  booktitle={Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers},
  pages={157--163},
  year={2017}
}

@article{artetxe2019massively,
  title={Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond},
  author={Artetxe, Mikel and Schwenk, Holger},
  journal={Transactions of the Association for Computational Linguistics},
  volume={7},
  pages={597--610},
  year={2019},
  publisher={MIT Press}
}

@inproceedings{liu2019variance,
  title={On the Variance of the Adaptive Learning Rate and Beyond},
  author={Liu, Liyuan and Jiang, Haoming and He, Pengcheng and Chen, Weizhu and Liu, Xiaodong and Gao, Jianfeng and Han, Jiawei},
  booktitle={International Conference on Learning Representations},
  year={2019}
}

@inproceedings{sennrich2016improving,
  title={Improving Neural Machine Translation Models with Monolingual Data},
  author={Sennrich, Rico and Haddow, Barry and Birch, Alexandra},
  booktitle={Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={86--96},
  year={2016}
}

@inproceedings{lakew2018comparison,
  title={A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation},
  author={Lakew, Surafel Melaku and Cettolo, Mauro and Federico, Marcello},
  booktitle={Proceedings of the 27th International Conference on Computational Linguistics},
  pages={641--652},
  year={2018}
}

@inproceedings{blackwood2018multilingual,
  title={Multilingual Neural Machine Translation with Task-Specific Attention},
  author={Blackwood, Graeme and Ballesteros, Miguel and Ward, Todd},
  booktitle={Proceedings of the 27th International Conference on Computational Linguistics},
  pages={3112--3122},
  year={2018}
}

@inproceedings{sachan2018parameter,
  title={Parameter Sharing Methods for Multilingual Self-Attentional Translation Models},
  author={Sachan, Devendra and Neubig, Graham},
  booktitle={Proceedings of the Third Conference on Machine Translation: Research Papers},
  pages={261--271},
  year={2018}
}

@inproceedings{zaremoodi2018adaptive,
  title={Adaptive knowledge sharing in multi-task learning: Improving low-resource neural machine translation},
  author={Zaremoodi, Poorya and Buntine, Wray and Haffari, Gholamreza},
  booktitle={Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  pages={656--661},
  year={2018}
}

@inproceedings{platanios2018contextual,
  title={Contextual Parameter Generation for Universal Neural Machine Translation},
  author={Platanios, Emmanouil Antonios and Sachan, Mrinmaya and Neubig, Graham and Mitchell, Tom},
  booktitle={Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  pages={425--435},
  year={2018}
}

@inproceedings{gu2018universal,
  title={Universal Neural Machine Translation for Extremely Low Resource Languages},
  author={Gu, Jiatao and Hassan, Hany and Devlin, Jacob and Li, Victor OK},
  booktitle={Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  pages={344--354},
  year={2018}
}

@inproceedings{dong2015multi,
  title={Multi-task learning for multiple language translation},
  author={Dong, Daxiang and Wu, Hua and He, Wei and Yu, Dianhai and Wang, Haifeng},
  booktitle={Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  pages={1723--1732},
  year={2015}
}

@inproceedings{zoph2016multi,
  title={Multi-Source Neural Translation},
  author={Zoph, Barret and Knight, Kevin},
  booktitle={Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  pages={30--34},
  year={2016}
}

@article{hochreiter1997long,
  title={Long short-term memory},
  author={Hochreiter, Sepp and Schmidhuber, J{\"u}rgen},
  journal={Neural computation},
  volume={9},
  number={8},
  pages={1735--1780},
  year={1997},
  publisher={MIT Press}
}
