  Most existing forecasting systems are memory-based methods, which attempt to
mimic human forecasting ability by employing various memory mechanisms and have
progressed in temporal modeling for memory dependency. Nevertheless, an obvious
weakness of this paradigm is that it can only model limited historical
dependence and can not transcend the past. In this paper, we rethink the
temporal dependence of event evolution and propose a novel
memory-anticipation-based paradigm to model an entire temporal structure,
including the past, present, and future. Based on this idea, we present
Memory-and-Anticipation Transformer (MAT), a memory-anticipation-based
approach, to address the online action detection and anticipation tasks. In
addition, owing to the inherent superiority of MAT, it can process online
action detection and anticipation tasks in a unified manner. The proposed MAT
model is tested on four challenging benchmarks TVSeries, THUMOS'14, HDD, and
EPIC-Kitchens-100, for online action detection and anticipation tasks, and it
significantly outperforms all existing methods. Code is available at
https://github.com/Echo0125/Memory-and-Anticipation-Transformer.
