  Neural attention has become central to many state-of-the-art models in
natural language processing and related domains. Attention networks are an
easy-to-train and effective method for softly simulating alignment; however,
the approach does not marginalize over latent alignments in a probabilistic
sense. This property makes it difficult to compare attention to other alignment
approaches, to compose it with probabilistic models, and to perform posterior
inference conditioned on observed data. A related latent approach, hard
attention, fixes these issues, but is generally harder to train and less
accurate. This work considers variational attention networks, alternatives to
soft and hard attention for learning latent variable alignment models, with
tighter approximation bounds based on amortized variational inference. We
further propose methods for reducing the variance of gradients to make these
approaches computationally feasible. Experiments show that for machine
translation and visual question answering, inefficient exact latent variable
models outperform standard neural attention, but these gains go away when using
hard attention based training. On the other hand, variational attention retains
most of the performance gain but with training speed comparable to neural
attention.
