  Despite the tremendous empirical success of neural models in natural language
processing, many of them lack the strong intuitions that accompany classical
machine learning approaches. Recently, connections have been shown between
convolutional neural networks (CNNs) and weighted finite state automata
(WFSAs), leading to new interpretations and insights. In this work, we show
that some recurrent neural networks also share this connection to WFSAs. We
characterize this connection formally, defining rational recurrences to be
recurrent hidden state update functions that can be written as the Forward
calculation of a finite set of WFSAs. We show that several recent neural models
use rational recurrences. Our analysis provides a fresh view of these models
and facilitates devising new neural architectures that draw inspiration from
WFSAs. We present one such model, which performs better than two recent
baselines on language modeling and text classification. Our results demonstrate
that transferring intuitions from classical models like WFSAs can be an
effective approach to designing and understanding neural models.
