  We present neural machine translation (NMT) models inspired by echo state
network (ESN), named Echo State NMT (ESNMT), in which the encoder and decoder
layer weights are randomly generated then fixed throughout training. We show
that even with this extremely simple model construction and training procedure,
ESNMT can already reach 70-80% quality of fully trainable baselines. We examine
how spectral radius of the reservoir, a key quantity that characterizes the
model, determines the model behavior. Our findings indicate that randomized
networks can work well even for complicated sequence-to-sequence prediction NLP
tasks.
