  Understanding the world around us and making decisions about the future is a
critical component to human intelligence. As autonomous systems continue to
develop, their ability to reason about the future will be the key to their
success. Semantic anticipation is a relatively under-explored area for which
autonomous vehicles could take advantage of (e.g., forecasting pedestrian
trajectories). Motivated by the need for real-time prediction in autonomous
systems, we propose to decompose the challenging semantic forecasting task into
two subtasks: current frame segmentation and future optical flow prediction.
Through this decomposition, we built an efficient, effective, low overhead
model with three main components: flow prediction network, feature-flow
aggregation LSTM, and end-to-end learnable warp layer. Our proposed method
achieves state-of-the-art accuracy on short-term and moving objects semantic
forecasting while simultaneously reducing model parameters by up to 95% and
increasing efficiency by greater than 40x.
