  Future predictions on sequence data (e.g., videos or audios) require the
algorithms to capture non-Markovian and compositional properties of high-level
semantics. Context-free grammars are natural choices to capture such
properties, but traditional grammar parsers (e.g., Earley parser) only take
symbolic sentences as inputs. In this paper, we generalize the Earley parser to
parse sequence data which is neither segmented nor labeled. This generalized
Earley parser integrates a grammar parser with a classifier to find the optimal
segmentation and labels, and makes top-down future predictions. Experiments
show that our method significantly outperforms other approaches for future
human activity prediction.
