  Semantic scene segmentation has primarily been addressed by forming
representations of single images both with supervised and unsupervised methods.
The problem of semantic segmentation in dynamic scenes has begun to recently
receive attention with video object segmentation approaches. What is not known
is how much extra information the temporal dynamics of the visual scene carries
that is complimentary to the information available in the individual frames of
the video. There is evidence that the human visual system can effectively
perceive the scene from temporal dynamics information of the scene's changing
visual characteristics without relying on the visual characteristics of
individual snapshots themselves. Our work takes steps to explore whether
machine perception can exhibit similar properties by combining appearance-based
representations and temporal dynamics representations in a joint-learning
problem that reveals the contribution of each toward successful dynamic scene
segmentation. Additionally, we provide the MIT Driving Scene Segmentation
dataset, which is a large-scale full driving scene segmentation dataset,
densely annotated for every pixel and every one of 5,000 video frames. This
dataset is intended to help further the exploration of the value of temporal
dynamics information for semantic segmentation in video.
