  Models of bags of words typically assume topic mixing so that the words in a
single bag come from a limited number of topics. We show here that many sets of
bag of words exhibit a very different pattern of variation than the patterns
that are efficiently captured by topic mixing. In many cases, from one bag of
words to the next, the words disappear and new ones appear as if the theme
slowly and smoothly shifted across documents (providing that the documents are
somehow ordered). Examples of latent structure that describe such ordering are
easily imagined. For example, the advancement of the date of the news stories
is reflected in a smooth change over the theme of the day as certain evolving
news stories fall out of favor and new events create new stories. Overlaps
among the stories of consecutive days can be modeled by using windows over
linearly arranged tight distributions over words. We show here that such
strategy can be extended to multiple dimensions and cases where the ordering of
data is not readily obvious. We demonstrate that this way of modeling
covariation in word occurrences outperforms standard topic models in
classification and prediction tasks in applications in biology, text modeling
and computer vision.
