  This paper proposes a novel algorithm which learns a formal regular grammar
from real-world continuous data, such as videos. Learning latent terminals,
non-terminals, and production rules directly from continuous data allows the
construction of a generative model capturing sequential structures with
multiple possibilities. Our model is fully differentiable, and provides easily
interpretable results which are important in order to understand the learned
structures. It outperforms the state-of-the-art on several challenging datasets
and is more accurate for forecasting future activities in videos. We plan to
open-source the code. https://sites.google.com/view/differentiable-grammars
