  Shadow removal in a single image has received increasing attention in recent
years. However, removing shadows over dynamic scenes remains largely
under-explored. In this paper, we propose the first data-driven video shadow
removal model, termed PSTNet, by exploiting three essential characteristics of
video shadows, i.e., physical property, spatio relation, and temporal
coherence. Specifically, a dedicated physical branch was established to conduct
local illumination estimation, which is more applicable for scenes with complex
lighting and textures, and then enhance the physical features via a mask-guided
attention strategy. Then, we develop a progressive aggregation module to
enhance the spatio and temporal characteristics of features maps, and
effectively integrate the three kinds of features. Furthermore, to tackle the
lack of datasets of paired shadow videos, we synthesize a dataset (SVSRD-85)
with aid of the popular game GTAV by controlling the switch of the shadow
renderer. Experiments against 9 state-of-the-art models, including image shadow
removers and image/video restoration methods, show that our method improves the
best SOTA in terms of RMSE error for the shadow area by 14.7. In addition, we
develop a lightweight model adaptation strategy to make our synthetic-driven
model effective in real world scenes. The visual comparison on the public
SBU-TimeLapse dataset verifies the generalization ability of our model in real
scenes.
