  Semantic segmentation has achieved great accuracy in understanding spatial
layout. For real-time tasks based on dynamic scenes, we extend semantic
segmentation in temporal domain to enhance the spatial accuracy with motion. We
utilize a shift-mode network over streaming input to ensure zero-latency
output. For the data overlap under shifting network, this paper identifies
repeated computation in fixed periods across network layers. To avoid this
redundancy, we derive a Shift-Memory Network (SMN) from encoding-decoding
baseline to reuse the network values without accuracy loss. Trained in
patch-mode, the SMN extracts the network parameters for SMN to perform
inference promptly in compact memory. We segment dynamic scenes from 1D
scanning input and 2D video. The experiments of SMN achieve equivalent accuracy
as shift-mode but in faster inference speeds and much smaller memory. This will
facilitate semantic segmentation in real-time application on edge devices.
