  Residual networks are an Euler discretization of solutions to Ordinary
Differential Equations (ODE). This paper explores a deeper relationship between
Transformer and numerical ODE methods. We first show that a residual block of
layers in Transformer can be described as a higher-order solution to ODE.
Inspired by this, we design a new architecture, {\it ODE Transformer}, which is
analogous to the Runge-Kutta method that is well motivated in ODE. As a natural
extension to Transformer, ODE Transformer is easy to implement and efficient to
use. Experimental results on the large-scale machine translation, abstractive
summarization, and grammar error correction tasks demonstrate the high
genericity of ODE Transformer. It can gain large improvements in model
performance over strong baselines (e.g., 30.77 and 44.11 BLEU scores on the
WMT'14 English-German and English-French benchmarks) at a slight cost in
inference efficiency.
